check for threads still running after program exits

Question

gcc 4.4.3 c89 pthreads

I use valgrind for checking memory errors.

I am just wondering if there is any tool for linux that can detect running threads that haven't been terminated after the program finishes.

I am running a multi-thread application and need a tool to make sure all threads have finished.

Many thanks for any suggestions,

Answer 1

If the program has terminated (because the initial thread returned from main(), some thread called exit(), or a fatal signal was recieved by the process) then you are guaranteed that all threads have been terminated with extreme prejudice.

---

If you want to write your program so that it ensures that all its threads have exited before main() exits, then you need to loop over all your threads at the end of main(), calling pthread_join() on each one. (This also means that you shouldn't create your threads detached, or detach them).

Answer 2

A Tool Approach

You can use Valgrind to help with this (via it's Helgrind tool), but it requires minor modification of the code. For each thread, you make the thread lock a unique mutex when the thread is created, and release the mutex when the thread exits. Then, when run under Helgrind, you will get a warning if the thread hasn't exited when the program terminates because the thread will still be holding the lock to the mutex. Consider this example thread start routine:

void * thread_start (void *arg)
{
    pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;

    pthread_mutex_lock(&mutex);

    // ...
    // Here the thread does whatever it normally does
    // ...

    // Unlock the mutex before exiting
    pthread_mutex_unlock(&mutex);
}

Simply run the program using Valgrind's Helgrind tool like so:

$ valgrind --tool=helgrind ./<program-name>

If the thread didn't exit when the program terminated, then Helgrind produces a warning like this:

==2203== Thread #2 was created
==2203==    at 0x31C96D3CDE: clone (in /lib64/libc-2.5.so)
==2203==    by 0x31CA206D87: pthread_create@@GLIBC_2.2.5 (in /lib64/libpthread-2.5.so)
==2203==    by 0x4A0B206: pthread_create_WRK (hg_intercepts.c:229)
==2203==    by 0x4A0B2AD: pthread_create@* (hg_intercepts.c:256)
==2203==    by 0x40060A: main (main.c:26)
==2203== 
==2203== Thread #2: Exiting thread still holds 1 lock
==2203==    at 0x4005DD: thread_start (main.c:13)
==2203==    by 0x4A0B330: mythread_wrapper (hg_intercepts.c:201)
==2203==    by 0x31CA20673C: start_thread (in /lib64/libpthread-2.5.so)
==2203==    by 0x31C96D3D1C: clone (in /lib64/libc-2.5.so)

You will get false positives using this method if you don't add the mutex unlock code anywhere the thread may exit (e.g. using pthread_exit), but fixing such a false-positive is easy once it is identified.

An Alternative Approach (Recommended)

Having said all of the above, that's probably not the approach I myself would take. Instead, I would write the program such that it cannot terminate until all threads have exited. The simplest way to achieve this is to call pthread_exit from the main thread before returning from main. Doing so will mean that the process will stay alive so long as any other thread is still running.

If you take this approach, and the process doesn't quit when you expect it to, then you know that a thread is still running. You can then attach a debugger to the process to determine which threads are still running and what they are doing.