subprocess.wait() not waiting for Popen process to finish (when using threads)?

Question

I am experiencing some problems when using subprocess.Popen() to spawn several instances of the same application from my python script using threads to have them running simultaneously. In each thread I run the application using the popen() call, and then I wait for it to finish by callingwait(). The problem seems to be that the wait()-call does not actually wait for the process to finish. I experimented by using only one thread, and by printing out text messages when the process starts, and when it finishes. So the thread function would look something like this:

def worker():
    while True:
        job = q.get() # q is a global Queue of jobs
        print('Starting process %d' % job['id'])
        proc = subprocess.Popen(job['cmd'], shell=True)
        proc.wait()
        print('Finished process %d' % job['id'])
        job.task_done()

But even when I only use one thread, it will print out several "Starting process..." messages, before any "Finished process..." message appears. Are there any cases when wait() does not actually wait? I have several different external applications (C++ console applications), which in turn will have several instances running simultaneously, and for some of them my code works, but for others it won't. Could there be some issue with the external applications that somehow affects the call to wait()? The code for creating the threads looks something like this:

for i in range(1):
    t = Thread(target=worker)
    t.daemon = True
    t.start()
q.join() # Wait for the queue to empty

Update 1: I should also add that for some of the external applications I sometimes get a return code (proc.returncode) of -1073471801. For example, one of the external applications will give that return code the first two times Popen is called, but not the last two (when I have four jobs).

Update 2: To clear things up, right now I have four jobs in the queue, which are four different test cases. When I run my code, for one of the external applications the first two Popen-calls generate the return code -1073471801. But if I print the exact command which Popen calls, and run it in a command window, it executes without any problems.

Solved! I managed to solve the issues I was having. I think the problem was my lack of experience in threaded programming. I missed the fact that when I had created my first worker threads, they would keep on living until the python script exits. By mistake I created more worker threads each time I put new items on the queue (I do that in batches for every external program I want to run). So by the time I got to the fourth external application, I had four threads running simultaneously even though I only thought I had one.

Answer 1

You could also use check_call() instead of Popen. check_call() waits for the command to finish, even when shell=True and then returns the exit code of the job.

Answer 2

Sadly when running your subprocess using shell=True, wait() will only wait for the sh subprocess to finish and not for the command cmd.

I will suggest if it possible to don't use the shell=True, if not possible you can create a process group like in this answer and use os.waitpid to wait for the process group not just the shell process.

Hope this was helpful :)