How do I make functions run as subprocesses in Python?

Question

I want my Python script to be able to run one of its functions as subprocesses. How should I do that?

Here is a mock-up script of my intention:

#!/urs/bin/env python

def print_mynumber(foo):
    """This function is obviously more complicated in my script.
    It should be run as a subprocess."""
    print(foo)

for foo in [1,2,3]:
    print_mynumber(foo) # Each call of this function should span a new process.
    # subprocess(print_mynumber(foo))

Thank you for your suggestions. It is a little hard for me to formulate the problem correctly, and thus to make the appropriate web search.

Answer 1

Use the multiprocessing module:

import multiprocessing as mp

def print_mynumber(foo):
    """This function is obviously more complicated in my script.
    It should be run as a subprocess."""
    print(foo)

if __name__ == '__main__':
    for foo in [1,2,3]:
        proc = mp.Process(target = print_mynumber, args = (foo, ))
        proc.start()

---

You might not want to be creating one process for each call to print_mynumber, especially if the list foo iterates over is long. A better way in that case would be to use a multiprocessing pool:

import multiprocessing as mp

def print_mynumber(foo):
    """This function is obviously more complicated in my script.
    It should be run as a subprocess."""
    print(foo)

if __name__ == '__main__':
    pool = mp.Pool()
    pool.map(print_mynumber, [1,2,3])

The pool, be default, will create N worker processes, where N is the number of cpus (or cores) the machine possesses. pool.map behaves much like the Python builtin map command, except that it farms out tasks to the pool of workers.