Is this a bug of httplib2

Question

My python program use httplib2.Http to make http request. Once I need to generate a request, I create a httplib2.Http object, so that my program will frequently create/destroy httplib2.Http objects.

I found my program was easy to crash due to reach max number of open files. Checking /proc//fd, there were too many open socket fds. The problem made me to have to dig into httplib2 source code.

Then I found that, in httplib2.Http._conn_request method, there was code like this:

        else:
            content = ""
            if method == "HEAD":
                conn.close()
            else:
                content = response.read()
            response = Response(response)
            if method != "HEAD":
                content = _decompressContent(response, content)
        break

This shows that socket is only closed when http method is HEAD. Maybe httplib2 wanted somehow to reuse the socket. But Http class doesn't have a close() method. That mean when I makes a Http request, the socket will not close until my process terminates.

Then I modified the code:

        else:
            content = ""
            if method == "HEAD":
                conn.close()
            else:
                content = response.read()
            response = Response(response)
            if method != "HEAD":
                content = _decompressContent(response, content)
                conn.close() # I ADD THIS CLOSE
        break

After that, my program worked well.

But I'm still curious if that's really httplib2's bug given that httplib2 is a very old and common lib.

Answer 1

No, the fact that the connections are not closed is not a bug, it is necessary to reuse a connection for the next request.

The way httplib2 is written, it waits for the server to close the connection or for it to time out, it doesn't actively close them (execept on errors). If a connection has been closed, httplib2 will just try to reestablish it the next time it's used.

Closing the connection (as your patch to the source does) would make this impossible, every request would have to use it's own connection.

Destroying the httplib2.Http object should automatically close the open connections as they are only referenced in it's connections dictionary, so if you don't keep any references to the Http objects you created it should work. Are you sure you don't have any references hanging around?

Alternatively you could pass a Connnection: close header to your request:

http.request("http://example.org", headers={'Connection': 'close'})

That should cause the server to close the connection after each request. Or, you could try to manually close all connections in the object's connections dict.