ZeroMQ bidirectional async communication with subprocesses

Question

I have a server process which receives requests from a web clients. The server has to call an external worker process ( another .py ) which streams data to the server and the server streams back to the client.

The server has to monitor these worker processes and send messages to them ( basically kill them or send messages to control which kind of data gets streamed ). These messages are asynchronous ( e.g. depend on the web client )

I thought in using ZeroMQ sockets over an ipc://-transport-class , but the call for socket.recv() method is blocking.

Should I use two sockets ( one for streaming data to the server and another to receive control messages from server )?

Answer 1

Using a separate socket for signalling and messaging is always better

While a Poller-instance will help a bit, the cardinal step is to use separate socket for signalling and another one for data-streaming. Always. The point is, that in such setup, both the Poller.poll() and the event-loop can remain socket-specific and spent not more than a predefined amount of time, during a real-time controlled code-execution.

So, do not hesitate to setup a bit richer signalling/messaging infrastructure as an environment where you will only enjoy the increased simplicity of control, separation of concerns and clarity of intents.

ZeroMQ is an excellent tool for doing this - including per-socket IO-thread affinity, so indeed a fine-grain performance tuning is available at your fingertips.