How to share data between multiple threads and processes in Python effectively?

Question

I have a main thread running Cmd from cmd2. This allows me to interactively start new threads using threading.Thread() performing simulations in "real-time". Every time step the simulation results are put() in a multiprocessing.Queue(). Additionally I can start live plots using matplotlib.animate. I read matplotlib is not thread-safe, so the plots run as a multiprocessing.Process() and get() the simulation results from the queue.

Unfortunately, once items from the queue are collected, they are deleted from the queue and not available for other threads or processes. This means I can send data from the simulation threads to the plotting processes, but can't use the simulation results at the same time in my main thread.

A solution could be to have two queues in each simulation thread: one queue to the main thread and one queue to the plotting process. This doesn't seem to be the optimal solution, but a rather complicated one.

Does anyone have an idea how to realize some kind of thread-safe shared variable every thread and processes can read from and write to?

Answer 1

In general, there are two main ways of sharing data between threads and memory:

• message passing
• shared memory

Python encourages you to avoid writing code that uses shared memory, and if you need to share data between threads and processes, you should just make a copy of the data and send it through a Queue.

If you need actual shared memory, this is fraught with many perils as you'll necessarily need to deal with locks to avoid problems. Additionally, this may not be possible with many python objects either as all python objects in a process shares a GIL.

I read matplotlib is not thread-safe

Code that aren't t multihread safe are usually not multiprocess safe either.

Does anyone have an idea how to realize some kind of thread-safe shared variable every thread and processes can read from and write to?

You don't want to, or if you really have to, use a database.