prevent unexpected stdin reads and lock in subprocess

Question

A simple case I'm trying to solve for all situations. I am running a subprocess for performing a certain task, and I don't expect it to ask for stdin, but in rare cases that I might not even expect, it might try to read. I would like to prevent it from hanging in that case.

here is a classic example:

import subprocess
p = subprocess.Popen(["unzip", "-tqq", "encrypted.zip"])
p.wait()

This will hang forever. I have already tried adding

stdin=open(os.devnull)

and such..

will post if I find a valuable solution. would be enough for me to receive an exception in the parent process - instead of hanging on communicate/wait endlessly.

update: it seems the problem might be even more complicated than I initially expected, the subprocess (in password and other cases) reads from other file descriptors - like the /dev/tty to interact with the shell. might not be as easy to solve as I thought..

Answer 1

If your child process may ask for a password then it may do it outside of standard input/output/error streams if a tty is available, see the first reason in Q: Why not just use a pipe (popen())?

As you've noticed, creating a new session prevents the subprocess from using the parent's tty e.g., if you have ask-password.py script:

#!/usr/bin/env python
"""Ask for password. It defaults to working with a terminal directly."""
from getpass import getpass

try:
    _ = getpass()
except EOFError:
    pass # ignore
else:
    assert 0

then to call it as a subprocess so that it would not hang awaiting for the password, you could use start_new_session=True parameter:

#!/usr/bin/env python3
import subprocess
import sys

subprocess.check_call([sys.executable, 'ask-password.py'],
                      stdin=subprocess.DEVNULL, start_new_session=True,
                      stderr=subprocess.DEVNULL)

stderr is redirected here too because getpass() uses it as a fallback, to print warnings and the prompt.

To emulate start_new_session=True on Unix on Python 2, you could use preexec_fn=os.setsid.

To emulate subprocess.DEVNULL on Python 2, you could use DEVNULL=open(os.devnull, 'r+b', 0) or pass stdin=PIPE and close it immediately using .communicate():

#!/usr/bin/env python2
import os
import sys
from subprocess import Popen, PIPE

Popen([sys.executable, 'ask-password.py'],
      stdin=PIPE, preexec_fn=os.setsid,
      stderr=PIPE).communicate() #NOTE: assume small output on stderr

Note: you don't need .communicate() unless you use subprocess.PIPE. check_call() is perfectly safe if you use an object with a real file descriptor (.fileno()) such as returned by open(os.devnull, ..). The redirection occurs before the child process is executed (after fork(), before exec()) -- there is no reason to use .communicate() instead of check_call() here.

Answer 2

Apparently the culprit is the direct usage of /dev/tty and such.

On linux at least, one solution is to add to the Popen call the following parameter:

preexec_fn=os.setsid

which causes a new session id to be set, and disallows reading from the tty directly. i will probably use the following code (stdin close is just in case):

import subprocess
import os
p = subprocess.Popen(["unzip", "-tqq", "encrypted.zip"],
                     stdin=subprocess.PIPE, preexec_fn=os.setsid)
p.stdin.close() #just in case
p.wait()

last two lines can be replaced by one call:

p.communicate()

since communicate() closes stdin file after sending all the input supplied.

Simple and elegant it seems.

Alternatively:

import subprocess
import os
p = subprocess.Popen(["unzip", "-tqq", "encrypted.zip"],
                     stdin=open(os.devnull), preexec_fn=os.setsid)
p.communicate()