Requests.get in Python using "User-Agent" not simulating a browser request

Question

I have to collect information from webpages using Python from a Linux terminal, it works wonderful but some pages (not all of them) are retrieving invalid URL's when I try to use requests.get due to they have agents detectors and they don't know how to answer my request (I'm not a browser or mobile application from a Linux terminal).

Using "User-Agent" header didn't work either, I tried several different ways to send it to emulate I am a Mozilla browser:

user_agent = {'User-Agent': 'Mozilla/5.0'}

or

user_agent = {'User-Agent': 'Mozilla/5.0 (Windows; U; Windows NT 5.1; hu-HU; rv:1.7.8) Gecko/20050511 Firefox/1.0.4'}

or many other combinations.

In some servers when I try to use this line:

page = requests.get(url, headers=user_agent)

I get a bad request, because these servers try to send me a webpage for desktop or mobile browsers and they fail to identify it.

Am I doing something wrong sending a User-Agent in this way? I tried my code in a Python Notebook and it works perfectly due to I'm currently (of course) sending a request from a browser.

Answer 1

You are using a very old user agent and indeed some sites will block you because of this.

>>> import requests
>>> header = {'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:32.0) Gecko/20100101 Firefox/32.0',}
>>> url = 'http://www.w3.org/'
>>> r = requests.get(url, headers=header)
>>> r.headers
CaseInsensitiveDict({'content-length': '40737', 'content-location': 'Home.html', 'accept-ranges': 'bytes', 'expires': 'Tue, 24 Jun 2014 04:44:36 GMT', 'vary': 'negotiate,accept', 'server': 'Apache/2', 'tcn': 'choice', 'last-modified': 'Mon, 23 Jun 2014 11:15:15 GMT', 'etag': '"9f21-4fc7ef51956c0;89-3f26bd17a2f00"', 'cache-control': 'max-age=600', 'date': 'Tue, 24 Jun 2014 04:34:36 GMT', 'p3p': 'policyref="http://www.w3.org/2001/05/P3P/p3p.xml"', 'content-type': 'text/html; charset=utf-8'})
>>> r.request.headers
CaseInsensitiveDict({'Accept-Encoding': 'gzip, deflate, compress', 'Accept': '*/*', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:32.0) Gecko/20100101 Firefox/32.0'})
>>>