Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

run multiple instances of python script simultaneously

I am trying to create 86 instances of task.py to run simultaneously.

import sys
import subprocess

for file in range(86):
    subprocess.call([sys.executable,'task.py',str(file)+'in.csv',str(filen)+'out.csv'])
like image 696
user14372 Avatar asked Oct 03 '13 10:10

user14372


People also ask

Can I run the same Python script simultaneously?

The simplest solution to run two Python processes concurrently is to run them from a bash file, and tell each process to go into the background with the & shell operator.

How do I run the same Python script in parallel?

We can also run the same function in parallel with different parameters using the Pool class. For parallel mapping, We have to first initialize multiprocessing. Pool() object. The first argument is the number of workers; if not given, that number will be equal to the number of elements in the system.

Can PyCharm run multiple scripts?

With PyCharm, you can run entire applications as well as particular scripts.


1 Answers

subprocess.call waits for command to complete. Use subprocess.Popen instead:

import sys
import subprocess

procs = []
for i in range(86):
    proc = subprocess.Popen([sys.executable, 'task.py', '{}in.csv'.format(i), '{}out.csv'.format(i)])
    procs.append(proc)

for proc in procs:
    proc.wait()
like image 104
falsetru Avatar answered Sep 19 '22 17:09

falsetru