How to run multiple commands synchronously from one subprocess.Popen command?

Question

Is it possible to execute an arbitrary number of commands in sequence using the same subprocess command?

I need each command to wait for the previous one to complete before executing and I need them all to be executed in the same session/shell. I also need this to work in Python 2.6, Python 3.5. I also need the subprocess command to work in Linux, Windows and macOS (which is why I'm just using echo commands as examples here).

See non-working code below:

import sys
import subprocess

cmds = ['echo start', 'echo mid', 'echo end']

p = subprocess.Popen(cmd=tuple([item for item in cmds]),
                     stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

for line in iter(p.stdout.readline, b''):
    sys.stdout.flush()
    print(">>> " + line.rstrip())

If this is not possible, which approach should I take in order to execute my commands in synchronous sequence within the same session/shell?

Answer 1

If you want to execute many commands one after the other in the same session/shell, you must start a shell and feed it with all the commands, one at a time followed by a new line, and close the pipe at the end. It makes sense if some commands are not true processes but shell commands that could for example change the shell environment.

Example using Python 2.7 under Windows:

encoding = 'latin1'
p = subprocess.Popen('cmd.exe', stdin=subprocess.PIPE,
             stdout=subprocess.PIPE, stderr=subprocess.PIPE)
for cmd in cmds:
    p.stdin.write(cmd + "\n")
p.stdin.close()
print p.stdout.read()

To have this code run under Linux, you would have to replace cmd.exe with /bin/bash and probably change the encoding to utf8.

For Python 3, you would have to encode the commands and probably decode their output, and to use parentheses with print.

Beware: this can only work for little output. If there was enough output to fill the pipe buffer before closing the stdin pipe, this code would deadlock. A more robust way would be to have a second thread to read the output of the commands to avoid that problem.