Detecting hangs with Python urllib2.urlopen

Question

I'm using Python's urllib2 to send an HTTP post:

import socket, urllib, urllib2

socket.setdefaulttimeout(15)    

postdata = urllib.urlencode({'value1' : 'a string', 'value2' : 'another string'})
headers = {
    'User-Agent': 'Agent',
    'Content-Type': 'application/x-www-form-urlencoded',
    'Accept': 'text/html, */*',
}

try: 
    request = urllib2.Request('http://www.example.com', postData, headers)
    response = urllib2.urlopen(request)
except urllib2.HTTPError, e:
    # Handle here
except urllib2.URLError, e:
    # Handle here
except httplib.HTTPException, e:
    # Handle here

Occasionally a network issue results in the call to urlopen never returning. We see other errors (including timeouts) handled correctly by the except block and have a call to socket.setdefaulttimeout() but there are still instances where the urlopen will never return.

I know it's never returning because we have some log lines in our actual code which get called before and after, and when this problem occurs only the calls before are made and the script hangs forever.

What's the best way to detect/handle this?

Answer 1

You can use signals, first set a handler for your signal

import signal
...
def handler(signum, frame):
    print 'Signal handler called with signal', signum
...
signal.signal(signal.SIGALRM, handler)

and put an alarm just before the urlopen call

signal.alarm(5)
response = urllib2.urlopen(request)
signal.alarm(0) # Disable the signal

after 5 seconds (or the time you desire) the OS will call the handler if the alarm is not disable (if urlopen never returns). More info about signal module: http://docs.python.org/library/signal.html