Executing same Python program with different arguments in parallel

Question

How can i execute the same python program in parallel (I am thinking of x10) with the only difference being two input arguments representing a time-range? I need it for some data-processing, which otherwise will take too long to finish

I know i can do it manually in shell by starting 10 scripts one by one, but it does not seem to be the most "elegant" solution + I would also love to define arguments for each of those programs dynamically in the "main" python program.

Is there a way to do it?

Answer 1

Enclose your script in a main method, like so:

def main(args):
    a, b = args
    # do something


if __name__ == '__main__':
    args = parse_arguments()
    main(args)

Then you can use a second script together with multiprocessing.Pool, to run the main method with different arguments.

from myscript import main
from multiprocessing import Pool



a = [1, 2, 3, 4, 5]
b = [6, 7, 8, 9, 10]

if __name__ == '__main__':
    with Pool(4) as pool: # four parallel jobs
        results = pool.map(main, zip(a, b))