Python and mongoDB connection pool ( pymongo )

Question

I have a web application, and there are thousands of requests every minutes. the following is my python code for mongoDB connection:

Tool.py:

globalconnection = None

def getCollection(name,safe=False,readpref=ReadPreference.PRIMARY):

    global globalconnection
    while globalconnection is None:
            try:
                    if not globalconnection is None:
                            globalconnection.close()
                    globalconnection = Connection('mongodb://host:port',replicaSet='mysetname',safe=False,read_preference=ReadPreference.PRIMARY,network_timeout=30,max_pool_size=1024)
            except Exception as e:
                    globalconnection = None

    request_context.connection = globalconnection

    return request_context.connection["mydb"]["mycoll"]

web.py

@app.route("/test")
def test():
    request_collection = getCollection("user")
    results = request_collection.find()
    for result in results:
        #do something...
        request_collection.save(result)
    request_collection.end_request()

One http request gets the connection through this function,

and the http request calls end_request before the end of the request.

But I found that there are many AutoReconnect errors and over 20000 connections in mongoDB while increasing requests.

Do you have any suggestion?

Answer 1

1. For auto-reconnection you simply catch the exception, and try to get the connection again: http://api.mongodb.org/python/current/api/pymongo/errors.html

2. 30 secs timeout sounds too long for, try shorter timeout instead?

3. Increase max number of connection from mongodb (default:20000) http://www.mongodb.org/display/DOCS/Connections