Mongo connections never released - Django and Mongoengine running on gunicorn with gevent

Question

I have a django application using mongoengine running on gunicorn with gevent workers. Under load, mongo connection count climbs up to about 3 thousand and never goes back down. Even after the load test is completed, the number of mongo connections stays constant. A restart of gunicorn releases the connections.

Package Versions

gunicorn==0.17.4
mongoengine==0.8.7
pymongo==2.7

mongodb 2.6.0

I have my mongoengine connection settings in an environment specific django settings file:

MONGO_DATABASES = {
    'default': {
        'DB': '****',
        'HOST': ***********:27017',
        'PORT': 27017
    }
}

from gevent import monkey
monkey.patch_all()
from mongoengine import connect
connect(MONGO_DATABASES['default']['DB'], host=MONGO_DATABASES['default']['HOST'],       port=MONGO_DATABASES['default']['PORT'], max_pool_size=100)

Is there something I need to do to make sure that unused connections eventually get released?

Thanks,

Doug

Answer 1

The feature you are looking for is a configuration option called maxIdleTimeMS, which is meant to be used for cleaning up after load spikes (among other things).

According to the docs:

maxIdleTimeMS - maximum time in millis a socket can sit idle in the pool before being closed and discarded. Useful for cleaning up after a load spike.

Sadly, this option is currently (March 2015) not available in the pymongo driver (which mongoengine uses under the hood), but it's being worked on! See the relevant JIRA ticket (and don't forget to upvote the issue!). It should be available with pymongo 3.1 in a few months.