How to avoid hanging wget in bash

Question

I have .sh script that, among other things, wgets currency exchange rates from Google as follows:

printf 'Bash: Going to get exchange rates'
echo    
wget -qO- "http://www.google.com/finance/converter?a=1&from=usd&to=aud" |  sed '/res/!d;s/<[^>]*>//g' > exrates
wget -qO- "http://www.google.com/finance/converter?a=1&from=usd&to=jpy" |  sed '/res/!d;s/<[^>]*>//g' >> exrates
wget -qO- "http://www.google.com/finance/converter?a=1&from=usd&to=hkd" |  sed '/res/!d;s/<[^>]*>//g' >> exrates
wget -qO- "http://www.google.com/finance/converter?a=1&from=usd&to=nzd" |  sed '/res/!d;s/<[^>]*>//g' >> exrates
wget -qO- "http://www.google.com/finance/converter?a=1&from=usd&to=eur" |  sed '/res/!d;s/<[^>]*>//g' >> exrates
wget -qO- "http://www.google.com/finance/converter?a=1&from=usd&to=gbp" |  sed '/res/!d;s/<[^>]*>//g' >> exrates

mv /home/stan/perl/2014/scripts/exrates /home/stan/perl/2014/exrates/exrates

printf 'Bash: Got exchange rates'
echo

Occasionally script hangs here, however. I don't mind not updating these rates every time it runs, if it hangs I'd like to skip this step altogether, but how?

What should I put in "if" statement to check if wget can fetch data promptly or will take forever? A little more verbosity in wget execution wouldn't hurt either.

Btw, I don't know why wget hangs. Browser opens those pages okay, and same commands run from terminal line by line work, too.

Answer 1

I assume that it hangs because you have a number of HTTP requests being sent to a single host in a script. The host in question doesn't like that too much and it starts to block requests from your IP address.

A simple workaround would be to put a sleep in between the requests. You could also make use of a function:

getExchangeRates() {
  wget -qO- "http://www.google.com/finance/converter?a=1&from=usd&to=$1" |  sed '/res/!d;s/<[^>]*>//g' >> exrates
  sleep 10   # Adding a 10 second sleep
}

and invoke it by passing a parameter to the function:

getExchangeRates aud

The function could also be invoked in a loop for various currencies:

for currency in aud jpy hkd nzd eur gpb; do
  getExchangeRates $currency
done

Answer 2

use timeout in wget statement only

wget --timeout 10 <URL>

timeout is in seconds and put some sleep in between two wgets