Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

python urllib2: connection reset by peer

Tags:

python

urllib2

I have a perl program that retrieves data from the database of my university library and it works well. Now I want to rewrite it in python but encounter the problem <urlopen error [errno 104] connection reset by peer>

The perl code is:

    my $ua = LWP::UserAgent->new;
    $ua->cookie_jar( HTTP::Cookies->new() );
    $ua->timeout(30);
    $ua->env_proxy;
    my $response = $ua->get($url); 

The python code I wrote is:

    cj = CookieJar();
    request = urllib2.Request(url); # url: target web page 
    opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj));
    opener = urllib2.install_opener(opener);
    data = urllib2.urlopen(request); 

I use VPN(virtual private network) to log in my university's library at home, and I tried both the perl code and python code. The perl code works as I expected, but the python code always encountered "urlopen error".

I googled for the problem and it seems that the urllib2 fails to load the environmental proxy. But according to the document of urllib2, the urlopen() function works transparently with proxies which do not require authentication. Now I feels quite confusing. Can anybody help me with this problem?

like image 745
hanqiang Avatar asked May 28 '11 18:05

hanqiang


People also ask

Is urllib2 deprecated?

urllib2 is deprecated in python 3. x. use urllib instaed.

What is the difference between Urllib and urllib2?

1) urllib2 can accept a Request object to set the headers for a URL request, urllib accepts only a URL. 2) urllib provides the urlencode method which is used for the generation of GET query strings, urllib2 doesn't have such a function. This is one of the reasons why urllib is often used along with urllib2.

Can I use urllib2 in python3?

NOTE: urllib2 is no longer available in Python 3 You can get more idea about urllib.

What does urllib2 do in Python?

Urllib package is the URL handling module for python. It is used to fetch URLs (Uniform Resource Locators). It uses the urlopen function and is able to fetch URLs using a variety of different protocols.


1 Answers

I tried faking the User-Agent headers as Uku Loskit and Mikko Ohtamaa suggested, and solved my problem. The code is as follows:

    proxy = "YOUR_PROXY_GOES_HERE"
    proxies = {"http":"http://%s" % proxy}
    headers={'User-agent' : 'Mozilla/5.0'}
    proxy_support = urllib2.ProxyHandler(proxies)
    opener = urllib2.build_opener(proxy_support, urllib2.HTTPHandler(debuglevel=1))
    urllib2.install_opener(opener)

    req = urllib2.Request(url, None, headers)
    html = urllib2.urlopen(req).read()
    print html

Hope it is useful for someone else!

like image 131
hanqiang Avatar answered Sep 22 '22 23:09

hanqiang