Correct incantation of subprocess with shell=True to get output and not hang

Question

Inside a subprocess call, I want to use shell=True so that it does globbing on pathnames (code below), however this has the annoying side-effect of making subprocess spawn a child process (which must then be `communicate()d/ poll()ed/ wait()ed/ terminate()d/ kill()ed/ whatevah).

(Yes I am aware the globbing can also be done with fnmatch/glob, but please show me the 'correct' use of subprocess on this, i.e. the minimal incantation to both get the stdout and stop the child process.)

This works fine (returns output):

subprocess.check_output(['/usr/bin/wc','-l','[A-Z]*/[A-Z]*.F*'], shell=False)

but this hangs

subprocess.check_output(['/usr/bin/wc','-l','[A-Z]*/[A-Z]*.F*'], shell=True)

(PS: It's seriously aggravating that you can't tell subprocess you want some but not all shell functionality e.g. globbing but not spawning. I think there's a worthy PEP in that, if anyone cares to comment, i.e. pass in a tuple of Boolean, or an or of binary flags)

(PPS: the idiom of whether you pass subprocess...(cmdstring.split() or [...]) is just a trivial idiomatic difference. I say tomato, you say tomay-to. In my case, the motivation is the command is fixed but I may want to call it more than once with a difference filespec.)

Answer 1

First off -- there's very little point to passing an array to:

subprocess.check_output(['/usr/bin/wc','-l','A-Z*/A-Z*.F*'], shell=True)

...as this simply runs wc with no arguments, in a shell also passed arguments -l and A-Z*/A-Z*.F* as arguments (to the shell, not to wc). Instead, you want:

subprocess.check_output('/usr/bin/wc -l A-Z*/A-Z*.F*', shell=True)

Before being corrected, this would hang because wc had no arguments and was reading from stdin. I would suggest ensuring that stdin is passed in closed, rather than passing along your Python program's stdin (as is the default behavior).

An easy way to do this, since you have shell=True:

subprocess.check_output(
    '/usr/bin/wc -l A-Z*/A-Z*.F* </dev/null',
    shell=True)

...alternately:

p = subprocess.Popen('/usr/bin/wc -l A-Z*/A-Z*.F*', shell=True,
                     stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=None)
(output, _) = p.communicate(input='')

...which will ensure an empty stdin from Python code rather than relying on the shell.