Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is python's "in" language construct thread-safe for lists?

Is obj in a_list thread-safe while a_list might be modified in a different thread?

Here's a comprehensive yet non-exhaustive list of examples of list operations and whether or not they are thread safe, however I couldn't find any reference for the in language construct.

In terms of python implementation, I use CPython, but answers re other implementations would be helpful too for the community.

like image 716
Jonathan Livni Avatar asked Nov 01 '13 13:11

Jonathan Livni


People also ask

Is list in Python thread-safe?

A list can be made thread-safe using a mutual exclusion (mutex) lock. Specifically, each add, delete, update, and read of the list must be protected by a lock.

Are Python dict threads safe?

Many common operations on a dict are atomic, meaning that they are thread-safe. Atomic means that the operation either occurs or does not occur with no in between inconsistent state. Operations such as adding, removing, and reading a value on a dict are atomic.

Is list thread-safe?

A thread-safe variant of ArrayList in which all mutative operations (e.g., add, set, remove..) are implemented by creating a separate copy of an underlying array. It achieves thread safety by creating a separate copy of the List which is different than vector or other collections used to provide thread-safety.

Is Python list pop thread-safe?

Any manipulation on list won't be atomic operation, so extra care need to be taken to make it thread safe using Lock, Event, Condition or Semaphores etc. This is explained in Are lists thread-safe here.


1 Answers

I am assuming you are using CPython here.

Provided there is no custom __contains__ or __iter__ hook dropping back into Python or the values you test against contained in the list use custom __eq__ hooks implemented in Python code, the in operator can be handled entirely in C, and is just one opcode.

This makes the operation entirely thread-safe; Python threads only switch between opcodes; the GIL (global interpreter lock) normally only unlocks between opcodes.

That said, if you use in on a custom C type that unlocks the GIL when testing containment would not be thread-safe.

In other words: the in bytecode test is locked, but if the operator needs to call Python code (through __contains__, iterating with __iter__ when no __contains__ implementation is available, or the values are test against __eq__ hooks implemented in Python), then the operation is not thread safe.

For other Python implementations, how threading is handled can vary widely. Certainly, Jython and IronPython have no GIL, and you should assume the operation is not thread safe.

like image 89
Martijn Pieters Avatar answered Oct 04 '22 07:10

Martijn Pieters