Execute a list of process without multiprocessing pool map

Question

import multiprocessing as mp

if __name__ == '__main__':

    #pool = mp.Pool(M)

    p1 = mp.Process(target= target1, args= (arg1,))
    p2 = mp.Process(target= target2, args= (arg1,))
    ...
    p9 = mp.Process(target= target9, args= (arg9,))
    p10 = mp.Process(target= target10, args= (arg10,))
    ...
    pN = mp.Process(target= targetN, args= (argN,))

    processList = [p1, p2, .... , p9, p10, ... ,pN]

I have N different target functions which consume unequal non-trivial amount of time to execute.

I am looking for a way to execute them in parallel such that M (1 < M < N) processes are running simultaneously. And as soon as a process is finished next process should start from the list, until all the processes in processList are completed.

As I am not calling the same target function, I could not use Pool.

I considered doing something like this:

    for i in range(0, N, M):
        limit = i + M
        if(limit > N):
            limit = N
        for p in processList[i:limit]:
            p.join()

Since my target functions consume unequal time to execute, this method is not really efficient.

Any suggestions? Thanks in advance.

EDIT: Question title has been changed to 'Execute a list of process without multiprocessing pool map' from 'Execute a list of process without multiprocessing pool'.

Answer 1

You can use proccess Pool:

#!/usr/bin/env python
# coding=utf-8

from multiprocessing import Pool
import random
import time


def target_1():
    time.sleep(random.uniform(0.5, 2))
    print('done target 1')

def target_2():
    time.sleep(random.uniform(0.5, 2))
    print('done target 1')

def target_3():
    time.sleep(random.uniform(0.5, 2))
    print('done target 1')

def target_4():
    time.sleep(random.uniform(0.5, 2))
    print('done target 1')


pool = Pool(2) # maximum two processes at time.
pool.apply_async(target_1)
pool.apply_async(target_2)
pool.apply_async(target_3)
pool.apply_async(target_4)
pool.close()
pool.join()

Pool is created specifically for what you need to do - execute many tasks in limited number of processes.

I also suggest you take a look at concurrent.futures library and it's backport to Python 2.7. It has a ProcessPoolExecutor, which has roughly same capabilities, but it's methods returns Future objects, and they has a nicer API.