Debug PyThread_acquire_lock deadlock

Question

I have a multithreaded application running in production environment that hangs at the random time with FUTEX_WAIT_PRIVATE state of all threads and gdb shows that all the threads are trying to make lock calling PyThread_acquire_lock. It is really massive application with tens of thousands code lines and I can not guess in what line this error is occurred. Can I debug this issue somehow? I can patch threading.Lock calling and log to file all locks are acquired/released in the application and then read this file in the case of error occurred again but I think that there are other python functions calling PyThread_acquire_lock. So how can I debug the issue? Maybe it is possible to "subscribe" on this C function call from Python and log all these calls?

Answer 1

You are in one step from the answer: attach with gdb to deadlocked process and use Python gdb extensions to examine lines being in a deadlock.

For gdb --version >= 7:

sudo apt install python2.7-dbg python3-dbg
sudo gdb /usr/bin/python[3] <pid_of_deadlocked_process>
(gdb) thread apply all py-list
(gdb) thread 2
(gdb) py-up
(gdb) py-print <lock_object>

References: https://docs.python.org/devguide/gdb.html, https://wiki.python.org/moin/DebuggingWithGdb