Run function with positional and optional arguments in parallel in python (follow up)

Question

This is a follow up question to: Python: How can I run python functions in parallel?

Minimal Working Example:

'''
Created on 06.05.2015
https://stackoverflow.com/questions/7207309/python-how-can-i-run-python-functions-in-parallel
'''
from multiprocessing import Process
import time

def runInParallel(*fns):
    proc = []
    for fn in fns:
        p = Process(target=fn)
        p.start()
        proc.append(p)
    for p in proc:
        p.join()

def func1():
    s=time.time()
    print 'func1: starting', s 
    for i in xrange(1000000000):
        if i==i:
            pass
    e = time.time()
    print 'func1: finishing', e
    print 'duration', e-s

if __name__ == '__main__':
    s =time.time()
    runInParallel(func1, func1, func1, func1, func1)
    print time.time()-s

Which leeds to this (and it's exactly what i want):

func1: starting 1430920678.09

func1: starting 1430920678.53

func1: starting 1430920679.02

func1: starting 1430920679.57

func1: starting 1430920680.55

func1: finishing 1430920729.68

duration 51.1449999809

func1: finishing 1430920729.78

duration 51.6889998913

func1: finishing 1430920730.69

duration 51.1239998341

func1: finishing 1430920748.64

duration 69.6180000305

func1: finishing 1430920749.25

duration 68.7009999752

71.5629999638

However, my function has quite a load of arguments, so i tested it like this:

-> func1(a) now gets an argument passed.

'''
Created on 06.05.2015
https://stackoverflow.com/questions/7207309/python-how-can-i-run-python-functions-in-parallel
'''
from multiprocessing import Process
import time

def runInParallel(*fns):
    proc = []
    for fn in fns:
        p = Process(target=fn)
        p.start()
        proc.append(p)
    for p in proc:
        p.join()

def func1(a):
    s=time.time()
    print 'func1: starting', s 
    for i in xrange(a):
        if i==i:
            pass
    e = time.time()
    print 'func1: finishing', e
    print 'duration', e-s

if __name__ == '__main__':
    s =time.time()
    g=s
    runInParallel(func1(1000000000), func1(1000000000),
                  func1(1000000000), func1(1000000000),
                  func1(1000000000))
    print time.time()-s

So now this happens:

func1: starting 1430921299.08

func1: finishing 1430921327.84

duration 28.760999918

func1: starting 1430921327.84

func1: finishing 1430921357.68

duration 29.8410000801

func1: starting 1430921357.68

func1: finishing 1430921387.14

duration 29.4619998932

func1: starting 1430921387.14

func1: finishing 1430921416.52

duration 29.3849999905

func1: starting 1430921416.52

func1: finishing 1430921447.39

duration 30.864000082

151.392999887

The process is now sequential and no longer parallel, and i don't get why! What am I missing and doing wrong?

EDIT: Additionally, how would an example look like, whre a few arguments are positional and others which are optional?

Answer 1

You have to pass your arguments to the Process using the argument args. For instance:

def runInParallel(*fns):
    proc = []
    for fn, arg in fns:
        p = Process(target=fn, args=(arg,))
        p.start()
        proc.append(p)
    for p in proc:
        p.join()

And then call the function using:

runInParallel((func1, 10**9),
              (func1, 10**9),
              (func1, 10**9))

Also, you might consider using a Pool instead:

from multiprocessing import Pool

pool = Pool()
pool.apply_async(func1, (10**9,))
pool.apply_async(func1, (10**9,))
pool.apply_async(func1, (10**9,))

EDIT:

Process and Pool.apply_asynch work the same way. They take two optional arguments args and kwargs. These are the standard variables for positional arguments and keyword arguments in python:

f(1, 2, a=3, b=4)  # is equivalent to
args, kwargs = (1, 2), {"a":3, "b":4}
f(*args, **kwargs)

Same example with multiprocessing:

args, kwargs = (1, 2), {"a":3, "b":4}
Process(target=f, args=args, kwargs=kwargs).start()
# Or
pool = Pool()
args, kwargs = (1, 2), {"a":3, "b":4}
pool.apply_async(f, args, kwargs)