How do I get improved pymongo performance using threading?

Question

I'm trying to see performance improvements on pymongo, but I'm not observing any.

My sample db has 400,000 records. Essentially I'm seeing threaded and single threaded performance be equal - and the only performance gain coming from multiple process execution.

Does pymongo not release the GIL during queries?

Single Perf: real 0m0.618s

Multiproc:real 0m0.144s

Multithread:real 0m0.656s

Regular code:

choices = ['foo','bar','baz']


def regular_read(db, sample_choice):
    rows = db.test_samples.find({'choice':sample_choice})
    return 42  # done to remove calculations from the picture

def main():
    client = MongoClient('localhost', 27017)
    db = client['test-async']
    for sample_choice in choices:
        regular_read(db, sample_choice)

if __name__ == '__main__':
    main()

$ time python3 mongotest_read.py 

real    0m0.618s
user    0m0.085s
sys 0m0.018s

Now when I use multiprocessing I can see some improvement.

from random import randint, choice

import functools
from pymongo import MongoClient
from concurrent import futures

choices = ['foo','bar','baz']
MAX_WORKERS = 4

def regular_read(sample_choice):
    client = MongoClient('localhost', 27017,connect=False)
    db = client['test-async']
    rows = db.test_samples.find({'choice':sample_choice})
    #return sum(r['data'] for r in rows)
    return 42

def main():
    f = functools.partial(regular_read)
    with futures.ProcessPoolExecutor(MAX_WORKERS) as executor:
        res = executor.map(f, choices)

    print(list(res))
    return len(list(res))

if __name__ == '__main__':
    main()

$ time python3 mongotest_proc_read.py 
[42, 42, 42]

real    0m0.144s
user    0m0.106s
sys 0m0.041s

But when you switch from ProcessPoolExecutor to ThreadPoolExecutor the speed drops back to single threaded mode.

...

def main():
    client = MongoClient('localhost', 27017,connect=False)
    f = functools.partial(regular_read, client)
    with futures.ThreadPoolExecutor(MAX_WORKERS) as executor:
        res = executor.map(f, choices)

    print(list(res))
    return len(list(res))

$ time python3 mongotest_thread_read.py 
[42, 42, 42]

real    0m0.656s 
user    0m0.111s
sys 0m0.024s

...

Answer 1

PyMongo uses the standard Python socket module, which does drop the GIL while sending and receiving data over the network. However, it's not MongoDB or the network that's your bottleneck: it's Python.

CPU-intensive Python processes do not scale by adding threads; indeed they slow down slightly due to context-switching and other inefficiencies. To use more than one CPU in Python, start subprocesses.

I know it doesn't seem intuitive that a "find" should be CPU intensive, but the Python interpreter is slow enough to contradict our intuition. If the query is fast and there's no latency to MongoDB on localhost, MongoDB can easily outperform the Python client. The experiment you just ran, substituting subprocesses for threads, confirms that Python performance is the bottleneck.

To ensure maximum throughput, make sure you have C extensions enabled: pymongo.has_c() == True. With that in place, PyMongo runs as fast as a Python client library can achieve, to get more throughput go to multiprocessing.

If your expected real-world scenario involves more time-consuming queries, or a remote MongoDB with some network latency, multithreading may give you some performance increase.