500 error with urllib.request.urlopen

Question

The following code:

   req = urllib.request.Request(url=r"http://borel.slu.edu/cgi-bin/cc.cgi?foirm_ionchur=im&foirm=Seol&hits=1&format=xml",headers={'User-Agent':' Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0'})
   handler = urllib.request.urlopen(req)

is giving me the following exception:

Traceback (most recent call last):
  File "C:/Users/Foo/lang/old/test.py", line 46, in <module>
    rip()
  File "C:/Users/Foo/lang/old/test.py", line 36, in rip
    handler = urllib.request.urlopen(req)
  File "C:\Python32\lib\urllib\request.py", line 138, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Python32\lib\urllib\request.py", line 375, in open
    response = meth(req, response)
  File "C:\Python32\lib\urllib\request.py", line 487, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Python32\lib\urllib\request.py", line 413, in error
    return self._call_chain(*args)
  File "C:\Python32\lib\urllib\request.py", line 347, in _call_chain
    result = func(*args)
  File "C:\Python32\lib\urllib\request.py", line 495, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 500: Internal Server Error

but it works fine in my browser, whats the issue?

Answer 1

The server is rather b0rken. It responds with a 500 error in the browser as well.

You can catch the exception and still read the response:

import urllib.request
from urllib.error import HTTPError

req = urllib.request.Request(url=r"http://borel.slu.edu/cgi-bin/cc.cgi?foirm_ionchur=im&foirm=Seol&hits=1&format=xml",headers={'User-Agent':' Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0'})
try:
    handler = urllib.request.urlopen(req)
except HTTPError as e:
    content = e.read()