Best Practice to Set Timeout for Python urllib.request.urlretrieve() – Python Web Crawler Tutorial

In most python application, if you plan to download some files, urllib.request.urlretrieve() will be your choice. The simplest reason is there is a callback function in it, we can show downloading progress bar in that callback function.

Syntax of urllib.request.urlretrieve()

urllib.request.urlretrieve(url, filename=None, reporthook=None, data=None)

However, we can not set timeout like urllib.request.urlopen() or OpenerDirector.open(). In this tutorial, we will introduce a simple way to set timeout for it.

From source code, we can find urllib.request.urlretrieve() will create a socket to connect and download file. We can set a global timeout for socket.

import socket
socket.setdefaulttimeout(time = 60) # 60 seconds

This timeout value will be urllib.request.urlretrieve() timeout.