File not found error when launching a subprocess containing piped commands

Question

I need to run the command date | grep -o -w '"+tz+"'' | wc -w using Python on my localhost. I am using subprocess module for the same and using the check_output method as I need to capture the output for the same.

However it is throwing me an error :

Traceback (most recent call last):   File "test.py", line 47, in <module>     check_timezone()   File "test.py", line 40, in check_timezone     count = subprocess.check_output(command)   File "/usr/lib/python2.7/subprocess.py", line 537, in check_output     process = Popen(stdout=PIPE, *popenargs, **kwargs)   File "/usr/lib/python2.7/subprocess.py", line 679, in __init__     errread, errwrite)   File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child     raise child_exception- OSError: [Errno 2] No such file or directory 

Answer 1

You have to add shell=True to execute a shell command. check_output is trying to find an executable called: date | grep -o -w '"+tz+"'' | wc -w and he cannot find it. (no idea why you removed the essential information from the error message).

See the difference between:

>>> subprocess.check_output('date | grep 1') Traceback (most recent call last):   File "<stdin>", line 1, in <module>   File "/usr/lib/python3.4/subprocess.py", line 603, in check_output     with Popen(*popenargs, stdout=PIPE, **kwargs) as process:   File "/usr/lib/python3.4/subprocess.py", line 848, in __init__     restore_signals, start_new_session)   File "/usr/lib/python3.4/subprocess.py", line 1446, in _execute_child     raise child_exception_type(errno_num, err_msg) FileNotFoundError: [Errno 2] No such file or directory: 'date | grep 1' 

And:

>>> subprocess.check_output('date | grep 1', shell=True) b'gio 19 giu 2014, 14.15.35, CEST\n' 

Read the documentation about the Frequently Used Arguments for more information about the shell argument and how it changes the interpretation of the other arguments.

---

Note that you should try to avoid using shell=True since spawning a shell can be a security hazard (even if you do not execute untrusted input attacks like Shellshock can still be performed!).

The documentation for the subprocess module has a little section about replacing the shell pipeline. You can do so by spawning the two processes in python and use subprocess.PIPE:

date_proc = subprocess.Popen(['date'], stdout=subprocess.PIPE) grep_proc = subprocess.check_output(['grep', '1'], stdin=date_proc.stdout, stdout=subprocess.PIPE) date_proc.stdout.close() output = grep_proc.communicate()[0] 

You can write some simple wrapper function to easily define pipelines:

import subprocess from shlex import split from collections import namedtuple from functools import reduce  proc_output = namedtuple('proc_output', 'stdout stderr')   def pipeline(starter_command, *commands):     if not commands:         try:             starter_command, *commands = starter_command.split('|')         except AttributeError:             pass     starter_command = _parse(starter_command)     starter = subprocess.Popen(starter_command, stdout=subprocess.PIPE)     last_proc = reduce(_create_pipe, map(_parse, commands), starter)     return proc_output(*last_proc.communicate())  def _create_pipe(previous, command):     proc = subprocess.Popen(command, stdin=previous.stdout, stdout=subprocess.PIPE)     previous.stdout.close()     return proc  def _parse(cmd):     try:         return split(cmd)     except Exception:         return cmd 

With this in place you can write pipeline('date | grep 1') or pipeline('date', 'grep 1') or pipeline(['date'], ['grep', '1'])

Answer 2

The most common cause of FileNotFound with subprocess, in my experience, is the use of spaces in your command. If you have just a single command (not a pipeline, and no redirection, wildcards, etc), use a list instead.

# Wrong, even with a valid command string subprocess.run(['grep -o -w "+tz+"'])  # Fixed; notice also  subprocess.run(["grep", "-o", "-w", '"+tz+"']) 

This change results in no more FileNotFound errors, and is a nice solution if you got here searching for that exception with a simpler command.

If you need a pipeline or other shell features, the simple fix is to add shell=True:

subprocess.run(     '''date | grep -o -w '"+tz+"'' | wc -w''',     shell=True) 

However, if you are using python 3.5 or greater, try using this approach:

import subprocess  a = subprocess.run(["date"], stdout=subprocess.PIPE) print(a.stdout.decode('utf-8'))  b = subprocess.run(["grep", "-o", "-w", '"+tz+"'],                    input=a.stdout, stdout=subprocess.PIPE) print(b.stdout.decode('utf-8'))  c = subprocess.run(["wc", "-w"],                    input=b.stdout, stdout=subprocess.PIPE) print(c.stdout.decode('utf-8')) 

You should see how one command's output becomes another's input just like using a shell pipe, but you can easily debug each step of the process in python. Using subprocess.run is recommended for python > 3.5, but not available in prior versions.