Python/Django "BadStatusLine" error

Question

I'm getting a weird error that I can't seem to find a solution for.

This error does not occur every time I hit this segment of code, and neither does it happen for the same iteration through the loop (it happens in a loop). If I run it enough, it doesn't seem to encounter the error and the program executes successfully. Regardless, I'd still like to figure out why this is happening.

Here is my error, versions, trace, etc: http://dpaste.com/681658/

It seems to happen with the following line in my code:

page = urllib2.urlopen(url)

Where url is.... a URL obviously.

And do have import urllib2 in my code.

Answer 1

The BadStatusLine exception is raised when you call urllib2.urlopen(url) and the remote server responds with a status code that python cannot understand.

Assuming that you don't control url, you can't prevent this from happening. All you can do is catch the exception, and manage it gracefully.

from httplib import BadStatusLine

try:
    page = urllib2.urlopen(url)
    # do something with page
except BadStatusLine:
    print "could not fetch %s" % url

Answer 2

This doesn't have anything to do with Django, it's an exception thrown by urllib2 which couldn't parse the response after fetching your url. It may be a network issue, a malformed response… Some servers / applications throw this kind of error randomly. If you don't control what this URL returns you're left with catching the exception, debugging which URLs are causing problems and trying to identify a pattern.

Answer 3

Explanations from other users are right and good, but in practice you may find this useful:
In my experience this usually happens when you are sending unquoted values to the url parameters, like values containing spaces or other characters that need to be quotes or url encoded.