When should I use subprocess.Popen instead of os.popen?

Question

Seems both executes a subprocess and create a pipe to do in/out, just that the subprocess is newer.

My question is, is there any function that subprocess.Popen can do while os.popen cannot, so that we need the new module subprocess?

Why Python language didn't choose to enhance os.popen but created a new module?

Answer 1

Short answer: Never use os.popen, always use subprocess!

As you can see from the Python 2.7 os.popen docs:

Deprecated since version 2.6: This function is obsolete. Use the subprocess module. Check especially the Replacing Older Functions with the subprocess Module section.

There were various limitations and problems with the old os.popen family of functions. And as the docs mention, the pre 2.6 versions weren't even reliable on Windows.

The motivation behind subprocess is explained in PEP 324 -- subprocess - New process module:

Motivation

Starting new processes is a common task in any programming language, and very common in a high-level language like Python. Good support for this task is needed, because:

• Inappropriate functions for starting processes could mean a security risk: If the program is started through the shell, and the arguments contain shell meta characters, the result can be disastrous. [1]

• It makes Python an even better replacement language for over-complicated shell scripts.

Currently, Python has a large number of different functions for process creation. This makes it hard for developers to choose.

The subprocess module provides the following enhancements over previous functions:

• One "unified" module provides all functionality from previous functions.

• Cross-process exceptions: Exceptions happening in the child before the new process has started to execute are re-raised in the parent. This means that it's easy to handle exec() failures, for example. With popen2, for example, it's impossible to detect if the execution failed.

• A hook for executing custom code between fork and exec. This can be used for, for example, changing uid.

• No implicit call of /bin/sh. This means that there is no need for escaping dangerous shell meta characters.

• All combinations of file descriptor redirection is possible. For example, the "python-dialog" [2] needs to spawn a process and redirect stderr, but not stdout. This is not possible with current functions, without using temporary files.

• With the subprocess module, it's possible to control if all open file descriptors should be closed before the new program is executed.

• Support for connecting several subprocesses (shell "pipe").

• Universal newline support.

• A communicate() method, which makes it easy to send stdin data and read stdout and stderr data, without risking deadlocks. Most people are aware of the flow control issues involved with child process communication, but not all have the patience or skills to write a fully correct and deadlock-free select loop. This means that many Python applications contain race conditions. A communicate() method in the standard library solves this problem.

Please see the PEP link for the Rationale, and further details.

Aside from the safety & reliability issues, IMHO, the old os.popen family was cumbersome and confusing. It was almost impossible to use correctly without closely referring to the docs while you were coding. In comparison, subprocess is a godsend, although it's still wise to refer to the docs while using it. ;)

Occasionally, one sees people recommending the use of os.popen rather than subprocess.Popen in Python 2.7, eg Python subprocess vs os.popen overhead because it's faster. Sure, it's faster, but that's because it doesn't do various things that are vital to guarantee that it's working safely!

---

FWIW, os.popen itself still exists in Python 3, however it's safely implemented via subprocess.Popen, so you might as well just use subprocess.Popen directly yourself. The other members of the os.popen family no longer exist in Python 3. The os.spawn family of functions still exist in Python 3, but the docs recommend that the more powerful facilities provided by the subprocess module be used instead.