Error codes returned by urllib/urllib2 and the actual page

Question

the normal behavior of urllib/urllib2 is if an error code is sent in the header of the response (i.e 404) an Exception is raised.

How do you look for specific errors i.e (40x, or 50x) based on the different errors, do different things. Also, how do you read the actual data being returned HTML/JSON etc (The data usually has error details which is different to the HTML error code)

Answer 1

urllib2 raises a HTTPError when HTTP errors happen. You can get to the response code using code on the exception object. You can get the response data using read():

>>> req = urllib2.Request('http://www.python.org/fish.html')
>>> try:
>>>     urllib2.urlopen(req)
>>> except urllib2.HTTPError, e:
>>>     print e.code
>>>     print e.read()
>>>
404
<actual data response will be here>

Answer 2

In urllib2 HTTPError exception is also a valid HTTP response, so you can treat an HTTP error as an exceptional event or valid response. But in urllib you have to subclass URLopener and define http_error_<code> method[s] or redefine http_error_default to handle them all.