Muti-core parallel computing over a for loop in python-3.x

Question

I have a simple for loop which is to print a number from 1 to 9999 with 5 seconds sleep in between. The code is as below:

import time

def run():
    length = 10000
    for i in range(1, length):
        print(i)
        time.sleep(5)
run()

I want to apply multiprocessing to run the for loop concurrently with multi-cores. So I amended the code above to take 5 cores:

import multiprocessing as mp
import time

def run():
    length = 10000
    for i in range(1, length):
        print(i)
        time.sleep(5)

if __name__ == '__main__':
        p = mp.Pool(5)
        p.map(run())
        p.close()

There is no issue in running the job but it seems like it is not running in parallel with 5 cores. How could I get the code worked as expected?

Answer 1

First, you are running the same 1..9999 loop 5 times, and second, you are executing the run function instead of passing it to the .map() method.

You must prepare your queue before passing it to the Pool instance so that all 5 workers process the same queue:

import multiprocessing as mp
import time

def run(i):
    print(i)
    time.sleep(5)

if __name__ == '__main__':
    length = 10000
    queue = range(1, length)
    p = mp.Pool(5)
    p.map(run, queue)
    p.close()

Note that it will process the numbers out of order as explained in the documentation. For example, worker #1 will process 1..500, worker #2 will process 501..1000 etc:

This method chops the iterable into a number of chunks which it submits to the process pool as separate tasks. The (approximate) size of these chunks can be specified by setting chunksize to a positive integer.

If you want to process the numbers more similarly to the single threaded version, set chunksize to 1:

p.map(run, queue, 1)