Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python equivalent of a given wget command

Tags:

python

wget

I'm trying to create a Python function that does the same thing as this wget command:

wget -c --read-timeout=5 --tries=0 "$URL" 

-c - Continue from where you left off if the download is interrupted.

--read-timeout=5 - If there is no new data coming in for over 5 seconds, give up and try again. Given -c this mean it will try again from where it left off.

--tries=0 - Retry forever.

Those three arguments used in tandem results in a download that cannot fail.

I want to duplicate those features in my Python script, but I don't know where to begin...

like image 778
Soviero Avatar asked Jun 21 '14 23:06

Soviero


People also ask

What is wget command in Python?

wget is a URL network downloader that can work in the background, and it helps in downloading files directly from the main server. In Python, this task is done by using the wget module.

Does Python have wget?

Downloading multiple files from the Internet manually as part of your daily routine can truly be a nightmare. And if you're looking for a way to automate your file downloads, then Python's Wget is the right tool for you.

How do I run wget in Python?

To run Wget commands from within a Python script, you'll use the Popen method of the subprocess package. Every time your script invokes popen() , it will execute the command you passed in an independent instance of the operating system's command processor.


2 Answers

There is also a nice Python module named wget that is pretty easy to use. Keep in mind that the package has not been updated since 2015 and has not implemented a number of important features, so it may be better to use other methods. It depends entirely on your use case. For simple downloading, this module is the ticket. If you need to do more, there are other solutions out there.

>>> import wget >>> url = 'http://www.futurecrew.com/skaven/song_files/mp3/razorback.mp3' >>> filename = wget.download(url) 100% [................................................] 3841532 / 3841532> >> filename 'razorback.mp3' 

Enjoy.

However, if wget doesn't work (I've had trouble with certain PDF files), try this solution.

Edit: You can also use the out parameter to use a custom output directory instead of current working directory.

>>> output_directory = <directory_name> >>> filename = wget.download(url, out=output_directory) >>> filename 'razorback.mp3' 
like image 56
Blairg23 Avatar answered Sep 24 '22 06:09

Blairg23


urllib.request should work. Just set it up in a while(not done) loop, check if a localfile already exists, if it does send a GET with a RANGE header, specifying how far you got in downloading the localfile. Be sure to use read() to append to the localfile until an error occurs.

This is also potentially a duplicate of Python urllib2 resume download doesn't work when network reconnects

like image 42
Eugene K Avatar answered Sep 26 '22 06:09

Eugene K