Understanding preemptive multitasking with locks and the Python GIL?

Question

I'm reading through Grok The GIL and it has the following statement in the discussion about locking.

So long as no thread holds a lock while it sleeps, does I/O, or some other GIL-dropping operation, you should use the coarsest, simplest locks possible. Other threads couldn't have run in parallel anyway.

It comes just after a discussion about preemptive multitasking. What prevents the preemptive dropping of the GIL from happening while you have a lock? Or is that not what this statement is referring to?

Answer 1

I asked the author of the piece and it comes down to the difference between dropping the GIL because you are waiting on an external operation vs an internal preemtion: https://opensource.com/article/17/4/grok-gil#comment-136186

Hi! Nothing prevents a thread from preemptively dropping the GIL while it holds a lock. Let's call that Thread A, and let's say there's also a Thread B. If Thread A holds a lock and gets preempted, then maybe Thread B could run instead of Thread A.

If Thread B is waiting for the lock that Thread A is holding, then Thread B is not waiting for the GIL. In that case Thread A reacquires the GIL immediately after dropping it, and Thread A continues.

If Thread B is not waiting for the lock that Thread A is holding, then Thread B might acquire the GIL and run.

My point about coarse locks, however, is this: no two threads can ever execute Python in parallel, because of the GIL. So using fine-grained locks doesn't improve throughput. This is in contrast to a language like Java or C, where fine-grained locks allow greater parallelism, and therefore greater throughput.

I still needed some clarification, and he did confirm this:

If I'm understanding you correctly, the intent of the statement I referenced was to avoid using locks around external operations, where you could then block multiple threads, if they all depended on that lock.

For the preemptive example, Thread A isn't blocked by anything externally, so the processing just goes back and forth similar to cooperative multitasking.