How are daemon threads killed in python (2.7)

Question

What happens to daemon threads when the main thread exits? How are they shut down? Is there a way the ensure that cleanup code is executed?

Here's a sketch of the situation I have:

import os
import subprocess
import threading
import glob
import datetime

def do_big_job(result='result.txt'):
    tmpfile = result+'.tmp'
    err = subprocess.call(['my_time_consuming_script', '-o', tmpfile])
    if err == 0:
        os.rename(tmpfile, result)
    else:
        os.remove(tmpfile) # process failed, clean up temp file



def main():
    not_done = True
    update = 0
    while not_done:

        # we can see if new data are available,
        # and start a process to get/prep it
        if new_data_available():
            args = ('result{0:05}.txt'.format(update),)
            th = threading.Thread(target=do_big_job, args=args)
            update = update + 1
            th.daemon = True 
            th.start()

        # but we don't need/want to wait for that process
        # to complete before continuing to iterate

        currently_available = glob.glob('result*.txt')
        # <snip>
        # rest of loop just uses what is available,
        # we don't want/need to block for updated data
        # to meet other responsiveness requirements

I'd like to make sure that I don't leave any temporary files (or zombie processes) lying around if the main thread dies while one (or more) of the do_big_job threads are still running, but I also can't just set daemon=False since I cannot wait for them to complete when main exits.

Answer 1

When main thread exits all daemon threads exit too. So if your main() exits all daemon threads will exit too.

However, and coming to the second part of your question,according to the official python threading docs for I/O operations this may not happen gracefully.

Daemon threads are abruptly stopped at shutdown. Their resources (such as open files, database transactions, etc.) may not be released properly. If you want your threads to stop gracefully, make them non-daemonic and use a suitable signalling mechanism such as an Event.

So if you want to check whether there are any zombie threads lying around, you can call the isAlive() function. Now as to how to ensure that cleanup code is executed after the main thread dies; I think it depends on what you mean by dies.

If you mean normal exit of the script (e.g. KeyboardInterrupt exception, sys.exit() is called) then it's really worth taking a look at the atexit module, which registers functions to be executed upon the termination of a process.

If by "dying" you mean that main() is killed by a signal, or a Python fatal internal error occurs, or os._exit() is called then atexit module won't work. In that case, a possible solution that comes to my mind would be to create another watchdog process that is constantly monitoring your main() and will run the cleanup code when appropriate.