Concurrent access to list from multiple threads in python when data is appended constantly

Question

We have a list to which data is appended at regular time intervals and this procedure takes time so using usual mutex to protect the entire list during writes is not the most efficient solution. How to organize reads and writes to such list in a more concurrent fashion?

Answer 1

You don't need locking when using list.append() or list.extend() with multiple threads. These operations are thread-safe.

Here is a brief overview of operations that are thread-safe: https://docs.python.org/3/faq/library.html#what-kinds-of-global-value-mutation-are-thread-safe

It's also worth mentioning that from a performance standpoint it's much faster to prepare sub-lists in separate threads, and then extend the main list with these sub-lists.