Python Popen grep

Question

I'd like Popen to execute:

grep -i --line-buffered "grave" data/*.txt

When run from the shell, this gives me the wanted result. If I start, in the very same directory where I test grep, a python repl and follow the instruction from the docs, I obtain what should be the proper argument list to feed Popen with:

['grep', '-i', '--line-buffered', 'grave', 'data/*.txt']

The result of p = subprocess.Popen(args) is

grep: data/*.txt: No such file or directory

and if I try p = subprocess.Popen(args, shell=True), I get:

Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.

Any help on how to perform the wanted process? I'm on MacOS Lion.

Answer 1

If you type * in bash the shell expands it to the files in the given directory before executing the command. Python's Popen does no such thing, so what you're doing when you call Popen like that is telling grep there is a file called *.txt in the data directory, instead of all the .txt files in the data directory. That file doesn't exist and you get the expected error.

To solve this you can tell python to run the command through the shell by passing shell=True to Popen:

subprocess.Popen('grep -i --line-buffered grave data/*.txt', shell=True)

Which gets translated to:

subprocess.Popen(['/bin/sh', '-c', 'grep -i --line-buffered "grave" data/*.txt'])

As explained in the documentation of Popen.

You have to use a string instead of a list here, because you want to execute /bin/sh -c "grep -i --line-buffered "grave" data/*.txt" (N.B. quotes around the command, making it a single argument to sh). If you use a list this command is run: /bin/sh -c grep -i --line-buffered "grave" data/*.txt, which gives you the output of simply running grep.