Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Why does shell=True eat my subprocess.Popen stdout?

It seems that using shell=True in the first process of a chain somehow drops the stdout from downstream tasks:

p1 = Popen(['echo','hello'], stdout=PIPE)
p2 = Popen('cat', stdin=p1.stdout, stdout=PIPE)
p2.communicate()
# outputs correctly ('hello\n', None)

Making the first process use shell=True kills the output somehow...

p1 = Popen(['echo','hello'], stdout=PIPE, shell=True)
p2 = Popen('cat', stdin=p1.stdout, stdout=PIPE)
p2.communicate()
# outputs incorrectly ('\n', None)

shell=True on the second process doesn't seem to matter. Is this expected behavior?

like image 689
Jake Biesinger Avatar asked May 19 '12 00:05

Jake Biesinger


People also ask

How can we avoid shell true in subprocess?

From the docs: args is required for all calls and should be a string, or a sequence of program arguments. Providing a sequence of arguments is generally preferred, as it allows the module to take care of any required escaping and quoting of arguments (e.g. to permit spaces in file names).

What does shell true do in subprocess?

On POSIX with shell=True , the shell defaults to /bin/sh . If args is a string, the string specifies the command to execute through the shell. This means that the string must be formatted exactly as it would be when typed at the shell prompt.

What is Popen in subprocess?

The subprocess module defines one class, Popen and a few wrapper functions that use that class. The constructor for Popen takes arguments to set up the new process so the parent can communicate with it via pipes. It provides all of the functionality of the other modules and functions it replaces, and more.

How do I capture the output of a subprocess run?

To capture the output of the subprocess. run method, use an additional argument named “capture_output=True”. You can individually access stdout and stderr values by using “output. stdout” and “output.


1 Answers

When you pass shell=True, Popen expects a single string argument, not a list. So when you do this:

p1 = Popen(['echo','hello'], stdout=PIPE, shell=True)

What happens is this:

execve("/bin/sh", ["/bin/sh", "-c", "echo", "hello"], ...)

That is, it calls sh -c "echo", and hello is effectively ignored (technically it becomes a positional argument to the shell). So the shell runs echo, which prints \n, which is why you see that in your output.

If you use shell=True, you need to do this:

p1 = Popen('echo hello', stdout=PIPE, shell=True)
like image 188
larsks Avatar answered Nov 01 '22 15:11

larsks