Menu
I have a long-running Python service and I'd like to know how much cumulative wall clock time has been spent by any runnable threads (i.e., threads that weren't blocked for some other reason) waiting for the GIL. Is there an easy way to do this? E.g., perhaps I could periodically dump some counter to its log file.
My underlying motivation is to rule out the GIL as a source of mystery response latency from these long-running processes. There is no particular reason to suspect the GIL (other than that it would fit the symptoms), but other forms of logging haven't turned up anything yet, so, if it is easy, it would be nice to have this information.
248
Model #3: Non-Python code can explicitly release the GIL If we run time. sleep(3) , that will do nothing for 3 seconds. We saw above that long-running extension code can prevent the GIL from being automatically switched between threads.
The GIL is a single lock on the interpreter itself which adds a rule that execution of any Python bytecode requires acquiring the interpreter lock. This prevents deadlocks (as there is only one lock) and doesn't introduce much performance overhead. But it effectively makes any CPU-bound Python program single-threaded.
ParallelPython: if you really need to scale, ParallelPython provides a mechanism for parallel execution of python code for multiple cores and clusters. Use a different Python implementation (both Jython and IronPython run without a GIL and the PyPy STM branch may also work for your use case).
Don't expect Python 3.11 to drop the GIL just yet. Merging Sam's work back to CPython will itself be a laborious process, but is only part of what's needed: a very good backwards compatibility and migration plan for the community is needed before CPython drops the GIL. None of this is planned yet.
I don't think there's an easy way. There's probably an awkward way, involving rebuilding Python to traverse the PyThreadState list and count the threads each time the lock is acquired, but I doubt it's worth the effort!
I know this is a speculative question but if you are even moderately concerned about there being delays caused by threading it may be prudent to move to a multiprocessing model instead of a multithreading model. Since processes are both safer and more scalable in Python they are almost always the best choice if practical.
132
You can now profile GIL usage using gil_load.
See my answer to the to that other question.
26
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
