Multiprocessing vs gevent

Question

Currently I am using zeromq with pub-sub pattern, I have single worker to publish and many(8) subscriber (all will subscribe) to same pattern.

Now I tried multiprocessing to spawn subscribers it works. I am missing few messages.

Why I am using multiprocessing is to handle each message as it arrives and process them, every second publisher publishes 10 - 100 messages.

In this case it is advised use multiprocessing or gevent ?

Answer 1

Multiprocessing will clearly have much higher memory overhead but will utilize another core (and you don't have to worry about lack of pre-emption) so.. it depends on your needs. It's likely that multiple processes using gevent will get you the highest throughput / lowest latency.

Answer 2

I am answering my own question

I choose multiprocessing over gevent

1. Server had 8 cores
2. Parallelism was needed
3. Choose ZMQ over multiprocessing queue or gevent queue.