Ordering in process pool python

Question

I am trying to return the result from processes running in parallel in python. Here is the code .

from dateutil.parser import parse
from multiprocessing.dummy import Pool
from itertools import *
import time
import random

def abcd(a):
    time.sleep(a)
    print(a)
    return a

p = Pool(3)
a = p.starmap(abcd, zip([10, 2, 15]))
p.close()
p.join()
print(a)

Is the result always returned in the order of the array inside zip ?

Answer 1

Pool.starmap (and Pool.map) will return the values in order of the arguments passed into them -- this does not however mean that they will be run in order. If you don't care about order and are ok with unordered results (and potentially more performance) there is Pool.imap_unordered.

See the documentation

Answer 2

There is no guarantee that they will be in the same order because ultimately the OS handles scheduling the processes. They may end up in the same order, and you might be able to increase the chances of it if you set chunksize to 1 in starmap(). Otherwise, at best the results will be in execution order grouped by chunk size.

If you need to know the original order afterwards, the trick I use is to just pass in an index (or a numpy.ndindex for n-dimensional data) that is just re-emitted by the worker function.