Slow wget speeds when connecting to https pages

Question

I'm using wget to connect to a secure site like this:

wget -nc -i inputFile

where inputeFile consists of URLs like this:

https://clientWebsite.com/TheirPageName.asp?orderValue=1.00&merchantID=36&programmeID=92&ref=foo&Ofaz=0

This page returns a small gif file. For some reason, this is taking around 2.5 minutes. When I paste the same URL into a browser, I get back a response within seconds.

Does anyone have any idea what could be causing this?

The version of wget, by the way, is "GNU Wget 1.9+cvs-stable (Red Hat modified)"

Answer 1

I know this is a year old but this exact problem plagued us for days. Turns out it was our DNS server but I got around it by disabling IP6 on my box.

You can test it out prior to making the system change by adding "--inet4-only" to the end of the command (w/o quotes).

Answer 2

1. Try forging your UserAgent

   -U "Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-GB; rv:1.9.0.1) Gecko/2008070206 Firefox/3.0.1" 
   

2. Disable Ceritificate Checking ( slow )

   --no-check-certificate
   

3. Debug whats happening by enabling verbostity

   -v
   

4. Eliminate need for DNS lookups:

   Hardcode thier IP address in your HOSTS file

   /etc/hosts 
   123.122.121.120  foo.bar.com