The workers in ThreadPoolExecutor is not really daemon

Question

The thing I cannot figure out is that although ThreadPoolExecutor uses daemon workers, they will still run even if main thread exit.

I can provide a minimal example in python3.6.4:

import concurrent.futures import time   def fn():     while True:         time.sleep(5)         print("Hello")   thread_pool = concurrent.futures.ThreadPoolExecutor() thread_pool.submit(fn) while True:     time.sleep(1)     print("Wow") 

Both main thread and the worker thread are infinite loops. So if I use KeyboardInterrupt to terminate main thread, I expect that the whole program will terminate too. But actually the worker thread is still running even though it is a daemon thread.

The source code of ThreadPoolExecutor confirms that worker threads are daemon thread:

t = threading.Thread(target=_worker,                      args=(weakref.ref(self, weakref_cb),                            self._work_queue)) t.daemon = True t.start() self._threads.add(t) 

Further, if I manually create a daemon thread, it works like a charm:

from threading import Thread import time   def fn():     while True:         time.sleep(5)         print("Hello")   thread = Thread(target=fn) thread.daemon = True thread.start() while True:     time.sleep(1)     print("Wow") 

So I really cannot figure out this strange behavior.

Answer 1

Suddenly... I found why. According to much more source code of ThreadPoolExecutor:

# Workers are created as daemon threads. This is done to allow the interpreter # to exit when there are still idle threads in a ThreadPoolExecutor's thread # pool (i.e. shutdown() was not called). However, allowing workers to die with # the interpreter has two undesirable properties: #   - The workers would still be running during interpreter shutdown, #     meaning that they would fail in unpredictable ways. #   - The workers could be killed while evaluating a work item, which could #     be bad if the callable being evaluated has external side-effects e.g. #     writing to a file. # # To work around this problem, an exit handler is installed which tells the # workers to exit when their work queues are empty and then waits until the # threads finish.  _threads_queues = weakref.WeakKeyDictionary() _shutdown = False  def _python_exit():     global _shutdown     _shutdown = True     items = list(_threads_queues.items())     for t, q in items:         q.put(None)     for t, q in items:         t.join()  atexit.register(_python_exit) 

There is an exit handler which will join all unfinished worker...