How to register "atexit" function in python's multiprocessing subprocess?

Question

I have some subprocesses (using multiprocessing) and when they stop, each of them need do some final work. Something like the following, which did not work though...

import multiprocessing
import atexit

def final():
    print "final work"

def worker():
    print 'Doing some work'
    atexit.register(final)

if __name__ == '__main__':
    p = multiprocessing.Process(target=worker)
    p.start()
    p.join()

So how could I do this?

Answer 1

Processes started via multiprocessing do not exit normally and atexit functions get never called. You have to run cleanup stuff manually via a try/finally block, for example:

def worker():
    try:
        print('Doing some work')
    finally:
        final()

Or, if you really want to use atexit (see below for the drawbacks):

def worker():
    try:
        print('Doing some work')
    finally:
        atexit._run_exitfuncs()

---

Why don't subprocesses exit normally?

Technically, because they quit via os._exit(), skipping any cleanup job (including atexit functions, __del__() and weakref finalizers).

The reason behind this decision is that cleanup stuff risks being executed at the wrong time, in the wrong process. Most of the code that uses atexit (or other mechanisms) expects the code to be executed only once and only when the main interpreter exits (for example, one may use atexit to remove a temporary directory). By executing atexit functions every time a subprocess quits, you break this expectation and bad things may happen.

So, even though you can run atexit functions via atexit._run_exitfuncs(), avoid doing so, both because the function is undocumented and because you may not be the only user of atexit (other third party libraries may use it, and you may introduce bugs in their code).