Python - calling multiprocessing.pool inside a daemon

Question

I have a Python script which spawns a daemon process. Inside the process, I am using multiprocessing.pool to run 1 to 4 processes simultaneously.

When I run this outside the daemon process, it works perfectly (i.e., when I set run_from_debugger=True - see code below), but if I run the code via a daemon process, (i.e., run_from_debugger=False), async_function is never executed.

Is it possible to use multiprocessing.pool inside a daemon process??? I am using Python-daemon 1.6 as my daemon package (if it matters).

Code:

def loop_callback(params):
    #Spawn the process in the pool
    #  Because loop_callback is called many times, often faster than async_function executes,
    #  adding them to a pool allows for parallel execution.
    pool.apply_async(async_function, params)


def run_service():
    # loop is a method that can/will call loop_callback multiple times, and it will call
    #   loop_callback faster than the code in asyc_function executes
    loop(alignment_watch_folder, sleep_duration)


#Class declaration
app = App()

#Declare a pool of processes
#  processes=1 indicates serial execution
pool = Pool(processes=4)

#Either run from a daemon process or not
run_from_debugger = False

#Run the daemon process
if run_from_debugger:
    run_service()
else:
    daemon_runner = runner.DaemonRunner(app)
    daemon_runner.do_action()

Any advice would be greatly appreciated.

Answer 1

Quoting from the documentation of multiprocessing:

daemon

The process’s daemon flag, a Boolean value. This must be set before start() is called.

The initial value is inherited from the creating process.

When a process exits, it attempts to terminate all of its daemonic child processes.

Note that a daemonic process is not allowed to create child processes. Otherwise a daemonic process would leave its children orphaned if it gets terminated when its parent process exits. Additionally, these are not Unix daemons or services, they are normal processes that will be terminated (and not joined) if non-daemonic processes have exited.

Since multiprocessing.Pool has to create worker processes, you cannot daemonaize a process using it.