Python urllib2 timeout when using Tor as proxy?

Question

I am using Python's urllib2 with Tor as a proxy to access a website. When I open the site's main page it works fine but when I try to view the login page (not actually log-in but just view it) I get the following error...

URLError: <urlopen error (10060, 'Operation timed out')>

To counteract this I did the following:

import socket
socket.setdefaulttimeout(None).

I still get the same timeout error.

1. Does this mean the website is timing out on the server side? (I don't know much about http processes so sorry if this is a dumb question)
2. Is there any way I can correct it so that Python is able to view the page?

Thanks, Rob

Answer 1

According to the Python Socket Documentation the default is no timeout so specifying a value of "None" is redundant.

There are a number of possible reasons that your connection is dropping. One could be that your user-agent is "Python-urllib" which may very well be blocked. To change your user agent:

request = urllib2.Request('site.com/login')
request.add_header('User-Agent','Mozilla/5.0 (X11; U; Linux i686; it-IT; rv:1.9.0.2) Gecko/2008092313 Ubuntu/9.04 (jaunty) Firefox/3.5')

You may also want to try overriding the proxy settings before you try and open the url using something along the lines of:

proxy = urllib2.ProxyHandler({"http":"http://127.0.0.1:8118"})  
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)