ID of a Python thread as reported by top

Question

I am starting a bunch of different threads in my Python script. I want to keep track of the memory and CPU usage of each of these threads. I use top and ps -eLf for that.

But it turns out that the identifier returned by thread.start_new_thread() is different from the thread PID displayed by top and other similar programs. Is there a way to obtain this PID from with in the Python script? This way, I can determine which PID belongs to which thread.

Answer 1

Thanks to this post, I got the Python threads to report their respective thread IDs. First do a grep -r 'SYS_gettid' /usr/include/'. I got a line: #define SYS_gettid __NR_gettid Upon further grepping by grep -r '__NR_gettid' /usr/include/, I got a bunch of matching lines:

/usr/include/x86_64-linux-gnu/asm/unistd_32.h:#define __NR_gettid 224
/usr/include/x86_64-linux-gnu/asm/unistd_64.h:#define __NR_gettid 186
/usr/include/asm-generic/unistd.h:#define __NR_gettid 178

Now choose the one that matches your architecture. Mine was 186. Now include this code in all your Python thread scripts to get the thread ID as seen by the OS:

import ctypes
tid = ctypes.CDLL('libc.so.6').syscall(186)

Answer 2

Here's a patch to replace the python thread identifier with the TID as displayed in htop:

def patch_thread_identifier():
    """Replace python thread identifier by TID."""
    # Imports
    import threading, ctypes
    # Define get tid function
    def gettid():
        """Get TID as displayed by htop."""
        libc = 'libc.so.6'
        for cmd in (186, 224, 178):
            tid = ctypes.CDLL(libc).syscall(cmd)
            if tid != -1:
                return tid
    # Get current thread
    current = threading.current_thread()
    # Patch get_ident (or _get_ident in python 2)
    threading.get_ident = threading._get_ident = gettid
    # Update active dictionary
    threading._active[gettid()] = threading._active.pop(current.ident)
    # Set new identifier for the current thread
    current._set_ident()
    # Done
    print("threading._get_ident patched!")