Proper way of re-using and closing a subprocess object

Question

I have the following code in a loop:

while true:
    # Define shell_command
    p1 = Popen(shell_command, shell=shell_type, stdout=PIPE, stderr=PIPE, preexec_fn=os.setsid)
    result = p1.stdout.read(); 
    # Define condition
    if condition:
       break;

where shell_command is something like ls (it just prints stuff).

I have read in different places that I can close/terminate/exit a Popen object in a variety of ways, e.g. :

p1.stdout.close()
p1.stdin.close()
p1.terminate
p1.kill

My question is:

1. What is the proper way of closing a subprocess object once we are done using it?
2. Considering the nature of my script, is there a way to open a subprocess object only once and reuse it with different shell commands? Would that be more efficient in any way than opening new subprocess objects each time?

Update

I am still a bit confused about the sequence of steps to follow depending on whether I use p1.communicate() or p1.stdout.read() to interact with my process.

From what I understood in the answers and the comments:

If I use p1.communicate() I don't have to worry about releasing resources, since communicate() would wait until the process is finished, grab the output and properly close the subprocess object

If I follow the p1.stdout.read() route (which I think fits my situation, since the shell command is just supposed to print stuff) I should call things in this order:

1. p1.wait()
2. p1.stdout.read()
3. p1.terminate()

Is that right?

Answer 1

What is the proper way of closing a subprocess object once we are done using it?

stdout.close() and stdin.close() will not terminate a process unless it exits itself on end of input or on write errors.

.terminate() and .kill() both do the job, with kill being a bit more "drastic" on POSIX systems, as SIGKILL is sent, which cannot be ignored by the application. Specific differences are explained in this blog post, for example. On Windows, there's no difference.

Also, remember to .wait() and to close the pipes after killing a process to avoid zombies and force the freeing of resources.

A special case that is often encountered are processes which read from STDIN and write their result to STDOUT, closing themselves when EOF is encountered. With these kinds of programs, it's often sensible to use subprocess.communicate:

>>> p = Popen(["sort"], stdin=PIPE, stdout=PIPE)
>>> p.communicate("4\n3\n1")
('1\n3\n4\n', None)
>>> p.returncode
0

This can also be used for programs which print something and exit right after:

>>> p = Popen(["ls", "/home/niklas/test"], stdin=PIPE, stdout=PIPE)
>>> p.communicate()
('file1\nfile2\n', None)
>>> p.returncode
0

Considering the nature of my script, is there a way to open a subprocess object only once and reuse it with different shell commands? Would that be more efficient in any way than opening new subprocess objects each time?

I don't think the subprocess module supports this and I don't see what resources could be shared here, so I don't think it would give you a significant advantage.

Answer 2

Considering the nature of my script, is there a way to open a subprocess object only once and reuse it with different shell commands?

Yes.

#!/usr/bin/env python
from __future__ import print_function
import uuid
import random
from subprocess import Popen, PIPE, STDOUT

MARKER = str(uuid.uuid4())

shell_command = 'echo a'
p = Popen('sh', stdin=PIPE, stdout=PIPE, stderr=STDOUT,
          universal_newlines=True) # decode output as utf-8, newline is '\n'
while True:
    # write next command
    print(shell_command, file=p.stdin)
    # insert MARKER into stdout to separate output from different shell_command
    print("echo '%s'" % MARKER, file=p.stdin)
    # read command output
    for line in iter(p.stdout.readline, MARKER+'\n'):
        if line.endswith(MARKER+'\n'):
            print(line[:-len(MARKER)-1])
            break # command output ended without a newline
        print(line, end='')
    # exit on condition
    if random.random() < 0.1:
        break
# cleanup
p.stdout.close()
if p.stderr:
   p.stderr.close()
p.stdin.close()
p.wait()

Put while True inside try: ... finally: to perform the cleanup in case of exceptions. On Python 3.2+ you could use with Popen(...): instead.

Would that be more efficient in any way than opening new subprocess objects each time?

Does it matter in your case? Don't guess. Measure it.