Popen.communicate is stuck until process spawned by sub-process terminates

Question

I have the below three scripts, and when I run main.py file, it spawns child.py which again executes the subchild.py and terminates quicly, subchild.py however keeps executing for a lot of time.

The problem with this is, main.py is blocked at p.communicate() till subchild.py terminates. If I open task manager and kill the running subchild.py main.py immediately returns the output of child.py

So my questions are as below

• main.py is supposed to wait only till child.py terminates, why is it waiting for subchild.py to terminate ?
• How do I make the p.communicate() not wait till subchild.py completes its execution ?

    # main.py file

    if __name__ == '__main__':
        import subprocess
        p = subprocess.Popen(
            ["python", "child.py"],
            stdout=subprocess.PIPE,
            stdin=subprocess.PIPE,
            stderr=subprocess.PIPE
        )
        out, _ = p.communicate()
        print(out)


    # child.py file

    if __name__ == '__main__':
        import subprocess
        print("Child running")
        p = subprocess.Popen(["python", "subchild.py"])
        print("Child pid - ", p.pid)
        exit(1)


    # subchild.py file

    if __name__ == '__main__':
        import time
        time.sleep(10000)

Note: I'm trying this on Windows 7 Enterprise. I'm using python3.6.6

Update after comment: On the main.py I need child.py's pid, stdout, stderr and process object so that I can kill child.py from main.py at any later point if I want to. This code is a small snippet is part of some API which I am building where the user would want the control to kill the process if he wishes to. subprocesses.call or subprocesses.run would not let me get control over the process object. I also won't have control over what child.py command I will receive as input for main.py, So I need to somehow not wait for the subchild.py and exit immediately with child.py's output as soon as it completes.

Answer 1

Your communicate call doesn't just wait for the child to complete. It first tries to read the entire contents of the child's stdout and stderr.

Even when the child has completed, the parent can't stop reading, because the grandchild inherits the child's stdin, stdout, and stderr. The parent needs to wait for the grandchild to complete, or at least for the grandchild to close its stdout and stderr, before the parent can be sure it's done reading.