Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

wget force retry until there is a connection

Tags:

linux

bash

wget

I have a bash script that uses wget running on a 3g device. I want the script to pause until I get a response so I have setup my code like this:

wget -t 0 -T 15 --retry-connrefused www.example.com

The problem is that when there is no connection at all (3g drops for a few seconds) the DNS can't resolve the host and wget stops retrying.

Is there a way to force retry until the connection comes back? I know I can write a loop but I want to know if there is a setting in wget I can use. If there is not what the best variable to build the loop on?

More details:

With the code I wrote, wget will retry but if the device has no Internet connection at all (for example I pull out the 3g dongle from the device) it will stop retrying telling me it can't resolve the host address. It sends that after the 15 seconds defined by -T 15

wget: unable to resolve host address
like image 567
Mika Avatar asked Jun 22 '15 15:06

Mika


People also ask

Does wget have a timeout?

However, sometimes, wget receives no data from the server, in which case it'll wait. In the example above, if wget has waited for data longer than five seconds, it will abort and restart the download. The default read timeout is 900 seconds.

What is Spider mode in wget?

The wget tool is essentially a spider that scrapes / leeches web pages but some web hosts may block these spiders with the robots. txt files. Also, wget will not follow links on web pages that use the rel=nofollow attribute. You can however force wget to ignore the robots.

How many times does wget try?

The default is to retry 20 times, with the exception of fatal errors like "connection refused" or "not found" (404), which are not retried.

Can wget resume download?

To resume a wget download it's very straight forward. Open the terminal to the directory where you were downloading your file to and run wget with the -c flag to resume the download.


2 Answers

This loop should do this:

while true;do
wget -T 15 -c http://example.com && break
done


How it works:
  1. In case there is no network connection, the while loop will not break and it will run the wget command continuously and keep printing error message.
  2. As soon as it gets connected to the Internet, wget starts resolving the host and getting the files.
  3. Now if the connection is lost or some other error occurs, default retry (don't use 0 or inf i.e unlimited retry, use limited value) of wget will retry to get the files until timeout of 15 seconds reached. After 15 seconds the wget command will fail and print error output and thus the while loop won't break. So it will again reach in a state where there is no connection or such and keep printing error message.
  4. Again as soon as it gets connected or the error is resolved, wget starts resolving the host and getting the files. These steps (1-4) continue as long as the files are not downloaded completely.
  5. This wget command uses -c option, i.e resume option. So every instances of wget will start (downloading) from where it left off.
  6. When the files get downloaded completely and the wget command succeeds, the loop will break.
like image 172
Jahid Avatar answered Sep 22 '22 21:09

Jahid


Here is a script that you can use to resolve your problem

FILENAME=$1
DOWNURL=$2

wget -O "`echo $FILENAME`" "`echo $DOWNURL`"
FILESIZE=$(stat -c%s "$FILENAME")

while [ $FILESIZE \< 1000 ]; do
    sleep 3
    wget -O "`echo $FILENAME`" "`echo $DOWNURL`"
    FILESIZE=$(stat -c%s "$FILENAME")
done

The script forces the download to continue until it finishes. The 1000 could be changed to fit whatever size file you are downloading.

like image 38
jgr208 Avatar answered Sep 19 '22 21:09

jgr208