How to work with user-agent to download a webpage using Wget

Question

I am trying to download this page using Wget. Here is the page link:

http://cgi.ebay.com/ws/eBayISAPI.dll?ViewItem&rt=nc&item=250972882769&si=a8iGAIchyvEbn7KveYFZ5QbEE7o%3D&print=all&category=31387

And here is my cmd:

wget -O ebay.html --user-agent="Mozilla/5.0 (Windows NT 5.2; rv:2.0.1) Gecko/20100101 Firefox/4.0.1" "http://cgi.ebay.com/ws/eBayISAPI.dll?ViewItem&rt=nc&item=250972882769&si=a8iGAIchyvEbn7KveYFZ5QbEE7o%3D&print=all&category=31387"

When I use it to access the page using a browser it works fine. When I use Wget, it downloads another page, not the original one. I think the problem is for user-agent. What's the solution?

Answer 1

The problem isn't user-agent, it's a missing cookie or cookies. The solution is

1. Retrieve the normal product page with wget --save-cookies=ebay-cookies ,
2. Fish the "Print" link URL out of that HTML file. (I did this by hand, you should obviously write a script to do it.)
3. Retrieve the "Print" URL with wget --load-cookies=ebay-cookies

I tried it with a random product page; it worked.