Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Do threads in python need to be joined to avoid leakage?

I understand the purpose of joining a thread, I'm asking about resource use. My specific use-case here is that I have a long-running process that needs to spawn many threads, and during operation, checks if they have terminated and then cleans them up. The main thread waits on inotify events and spawns threads based on those, so it can't block on join() calls, because it needs to block on inotify calls.

I know that with pthreads, for instance, not joining a terminated thread will cause a resource leak:

PTHREAD_JOIN(3): Failure to join with a thread that is joinable (i.e., one that is not detached), produces a "zombie thread". Avoid doing this, since each zombie thread consumes some system resources, and when enough zombie threads have accumulated, it will no longer be possible to create new threads (or processes).

Python's documentation says no such thing, though, but it also doesn't specify that the join() can be disregarded without issue if many threads are expected to end on their own without being joined during normal operation.

I'm wondering, can I simply take my thread list and do the following:

threads = [thread for thread in threads if thread.is_alive()]

For each check, or will this leak? Or must I do the following?

alive_threads = list()
for thread in threads:
    if thread.is_alive():
        alive_threads.append(thread)
    else:
        thread.join()
threads = alive_threads
like image 211
Taywee Avatar asked Jul 08 '16 21:07

Taywee


1 Answers

TLDR: No. Thread cleans up the underlying resources by itself.


Thread.join merely waits for the thread to end, it does not perform cleanup. Basically, each Thread has a lock that is released when the thread is done and subsequently cleaned up. Thread.join just waits for the lock to be released.

There is some minor cleanup done by Thread.join, namely removing the lock and setting a flag to mark the thread as dead. This is an optimisation to avoid needlessly waiting for the lock. These are internal, however, and also performed by all other public methods relying on the lock and flag. Finally, this cleanup is functionally equivalent to a Thread being cleaned up automatically due to garbage collection.

like image 103
MisterMiyagi Avatar answered Oct 02 '22 12:10

MisterMiyagi