Pickling from multiple threads in Python

Question

I have a python program with multiple threads. Each thread detects events, which I would like to store somewhere so that I can read them in again (for testing). Right now, I'm using Pickle to output the events, and each thread outputs to a different file. Ideally, I would only use one output file, and all the threads would write to it, but when I try this, it looks like the various threads try to write their output at the same time, and they don't get pickled properly. Is there a way to do this?

Answer 1

seems like a good place to use a Queue.

• Have all your event detection threads put items on a shared Queue.
• Create another thread to get items from the queue, and write/pickle/whatever from this thread.

from the Queue docs:

"The Queue module implements multi-producer, multi-consumer queues. It is especially useful in threaded programming when information must be exchanged safely between multiple threads. The Queue class in this module implements all the required locking semantics. It depends on the availability of thread support in Python; see the threading module."