I/O error(socket error): [Errno 111] Connection refused

Question

I have a program that uses urllib to periodically fetch a url, and I see intermittent errors like :

I/O error(socket error): [Errno 111] Connection refused.

It works 90% of the time, but the othe r10% it fails. If retry the fetch immediately after it fails, it succeeds. I'm unable to figure out why this is so. I tried to see if any ports are available, and they are. Any debugging ideas?

For additional info, the stack trace is:

File "/usr/lib/python2.6/urllib.py", line 203, in open 
    return getattr(self, name)(url)

File "/usr/lib/python2.6/urllib.py", line 342, in open_http
    h.endheaders()

File "/usr/lib/python2.6/httplib.py", line 868, in endheaders
    self._send_output()

File "/usr/lib/python2.6/httplib.py", line 740, in _send_output
    self.send(msg)

File "/usr/lib/python2.6/httplib.py", line 699, in send
    self.connect()

File "/usr/lib/python2.6/httplib.py", line 683, in connect
    self.timeout)

File "/usr/lib/python2.6/socket.py", line 512, in create_connection
    raise error, msg

Edit - A google search isn't very helpful, what I got out of it is that the server I'm fetching from sometimes refuses connections, how can I verify its not a bug in my code and this is indeed the case?

Answer 1

Use a packet sniffer like Wireshark to look at what happens. You need to see a SYN-flagged packet outgoing, a SYN+ACK-flagged incoming and then a ACK-flagged outgoing. After that, the port is considered open on the local side.

If you only see the first packet and the error message comes after several seconds of waiting, the other side is not answering at all (like in: unplugged cable, overloaded server, misguided packet was discarded) and your local network stack aborts the connection attempt. If you see RST packets, the host actually denies the connection. If you see "ICMP Port unreachable" or host unreachable packets, a firewall or the target host inform you of the port actually being closed.

Of course you cannot expect the service to be available at all times (consider all the points of failure in between you and the data), so you should try again later.