best practice of django + PyMongo pooling?

Question

I have a db = pymongo.Connection() call in Django's views.py for a simple MongoDB connection to store some simple statistics.

What's the best practice to make it auto support MongoDB connection pooling?

Where do I need to put the end_request() code?

How do I choose the max_pool_size parameter during connection?

Answer 1

How does connection pooling work in PyMongo?

Every Connection instance has built-in connection pooling. By default, each thread gets its own socket reserved on its first operation. Those sockets are held until end_request() is called by that thread.

Calling end_request() allows the socket to be returned to the pool, and to be used by other threads instead of creating a new socket. Judicious use of this method is important for applications with many threads or with long running threads that make few calls to PyMongo operations.

Alternatively, a Connection created with auto_start_request=False will share sockets (safely) among all threads.

I think it comes down to the type of application you have and how long the requests will hold onto a connection. The idea of calling end_request helps with long running requests holding on to a socket for a long time and causing many sockets to get created. If a single request can release the connection when it no longer needs it, then the socket can be repurposed for other requests.

If they are fast requests, then I believe the auto_start_request=False works by reusing the socket.

Ensuring a connection keeps using the same socket means that is will have consistent reads. Think if you made a query but it got delayed, and then immeditely made another query and it used a different socket. This socket manages to respond before the previous. You would have inconsistent data since it does not reflect the previous write.