Wget 503 unavailable error

Question

$ wget www.amazon.com
Resolving www.amazon.com... 205.251.242.54
Connecting to www.amazon.com|205.251.242.54|:80... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2015-10-12 23:27:24 ERROR 503: Service Unavailable.

I am trying to issue wget on a URL and getting this error. I need to store the HTML files and I was hoping wget would work :(

I tried using the --no-proxy option but it doesn't help.

Answer 1

The problem is that amazon firewall is blocking connections whose user-agent is not set, or incorrect.

You may try setting user agent to wget and get amazon with following command (note that you may need to change a valid user agent if this one becomes invalid):

wget -U "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.79 Safari/537.36" https://amazon.com

Answer 2

I can issue wget http://www.amazon.com without any problem.
My guess is that you got blocked by amazon after scrapping a little bit to much...

503 Service Unavailable

The server is currently unable to handle the request due to a temporary overloading or maintenance of the server. The implication is that this is a temporary condition which will be alleviated after some delay. If known, the length of the delay MAY be indicated in a Retry-After header. If no Retry-After is given, the client SHOULD handle the response as it would for a 500 response.

  Note: The existence of the 503 status code does not imply that a
  server must use it when becoming overloaded. Some servers may wish
  to simply refuse the connection.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html