How can I wait for child processes?

Question

I have a Python script that starts tasks like this:

import os
os.system("./a.sh")
do_c()

But a.sh is a bash script that starts other programs. The bash script itself seems to be ready before all scripts that are started are ready.

How can I wait for all scripts (child processes) to be ready, before do_c() gets executed?

Clarification: When I write ready, I mean finish / exit.

Example

run.py

This file can be changed. But don't rely on sleep, as I don't know how long a.py and b.py take.

#!/usr/bin/env python

import os
from time import sleep

print("Started run.py")
os.system("./a.py")
print("a is ready.")
print("Now all messages should be there.")

sleep(30)

a.py

This may not be modified:

#!/usr/bin/env python

import subprocess
import sys

print("  Started a.py")
pid = subprocess.Popen([sys.executable, "b.py"])
print("  End of a.py")

b.py

This may not be modified:

#!/usr/bin/env python

from time import sleep

print("    Started b.py")
sleep(10)
print("    Ended b.py")

Desired output

The last message has to be Now all messages should be there..

Current output

started run.py
  Started a.py
  End of a.py
a is ready.
Now all messages should be there.
    Started b.py
    Ended b.py

Answer 1

The usual approaches for handling this kind of situations don't work. Waiting for a.py (which os.system does by default) doesn't work because a.py exits before its children are done executing. Finding the PID of b.py is tricky because, once a.py exits, b.py can no longer be connected to it in any way - even the parent PID of b.py is 1, the init process.

However, it is possible to make use of inherited file descriptor as a poor man's signal that a child is dead. Set up a pipe whose read end is in run.py, and whose write end is inherited by a.py and all its children. Only when the last child exits will the write-end of the pipe be closed, and a read() on the read-end of the pipe will cease to block.

Here is a modified version of run.py that implements this idea, displaying the desired output:

#!/usr/bin/env python

import os
from time import sleep

print("Started run.py")

r, w = os.pipe()
pid = os.fork()
if pid == 0:
    os.close(r)
    os.execlp("./a.py", "./a.py")
    os._exit(127)   # unreached unless execlp fails
os.close(w)
os.waitpid(pid, 0)  # wait for a.py to finish
print("a is ready.")

os.read(r, 1)       # wait for all the children that inherited `w` to finish
os.close(r)
print("Now all messages should be there.")

---

Explanation:

Pipe is an inter-process communication device that allows parent and child processes to communicate through inherited file descriptors. Normally one creates a pipe, fork a process, possibly executes an external file, and reads some data from the read-end of the pipe, the same data written by another to the write-end of the pipe. (Shells implement pipelines using this mechanism by going one step further and making the standard file descriptors such as stdin and stdout point to the appropriate ends of a pipe.)

In this case, we don't care about exchanging actual data with the children, we only want to be notified when they exit. To achieve that, we make use of the fact that when a process dies, the kernel closes all of its file descriptors. In turn, when a forked process inherits file descriptors, the file descriptor is considered closed when all the copies of the descriptor are closed. So we set up a pipe with a write-end that will get inherited by all the processes spawned by a.py. These processes don't need to know anything about this file descriptor, the only important thing is that when they all die, the write-end of the pipe will close. This will indicate at the read-end of the pipe by os.read() no longer blocking and returning a 0-length string that signals the end-of-file condition.

The code is a simple implementation of that idea:

• The part between os.pipe() and the first print is an implementation of os.system(), with the difference that it closes the read-end of the pipe in the child. (This is necessary — simply calling os.system() would keep the read-end open which would prevent the final read in the parent from working correctly.)

• os.fork() duplicates the current process, with the only way to differentiate the parent and the child being that in the parent you get the child PID (and the child gets 0, since it can always find out its PID using os.getpid()).

• The if pid == 0: branch runs in the child, and only execs ./a.py. "Exec" means that it runs the specified executable without ever returning. The os._exit() is only there in case execlp fails (probably unnecessary in Python because failure of execlp would raise an exception which would exit the program, but still). The rest of the program runs in the parent.

• The parent closes the write-end of the pipe (otherwise attempting to read from the read-end would deadlock). os.waitpid(pid) is the waiting for a.py normally performed by os.system(). In our case it's not necessary to call waitpid, but it's a good idea to do so to prevent a zombie from remaining.

• os.read(r, 1) is where the magic happens: it attempts to read at most 1 character from the read end of the pipe. Since no one ever writes to the write-end of the pipe, reading will block until the write-end of the pipe is closed. Since the children of a.py know nothing of the inherited file descriptor, the only way for it to be closed is by the kernel doing it after the death of the respective processes. When all of the inherited write-end descriptors are closed, os.read() returns a zero-length string, which we ignore and proceed with the execution.

• Finally, we close the write-end of the pipe, so that the shared resource is freed.