Menu
I'm trying to extract the response header of a URL request. When I use firebug to analyze the response output of a URL request, it returns:
Content-Type text/html However when I use the python code:
urllib2.urlopen(URL).info() the resulting output returns:
Content-Type: video/x-flv I am new to python, and to web programming in general; any helpful insight is much appreciated. Also, if more info is needed please let me know.
Thanks in advance for reading this post
267
urllib2 is a Python module that can be used for fetching URLs. It defines functions and classes to help with URL actions (basic and digest. authentication, redirections, cookies, etc) The magic starts with importing the urllib2 module.
urllib2 is deprecated in python 3. x. use urllib instaed.
The data returned by urlopen() or urlretrieve() is the raw data returned by the server. This may be binary data (such as an image), plain text or (for example) HTML. The HTTP protocol provides type information in the reply header, which can be inspected by looking at the Content-Type header.
True, if you want to avoid adding any dependencies, urllib is available. But note that even the Python official documentation recommends the requests library: "The Requests package is recommended for a higher-level HTTP client interface."
Try to request as Firefox does. You can see the request headers in Firebug, so add them to your request object:
import urllib2 request = urllib2.Request('http://your.tld/...') request.add_header('User-Agent', 'some fake agent string') request.add_header('Referer', 'fake referrer') ... response = urllib2.urlopen(request) # check content type: print response.info().getheader('Content-Type') There's also HTTPCookieProcessor which can make it better, but I don't think you'll need it in most cases. Have a look at python's documentation:
http://docs.python.org/library/urllib2.html
121
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
