Why does popen() invoke a shell to execute a process?

Question

I'm currently reading up on and experimenting with the different possibilities of running programs from within C code on Linux. My use cases cover all possible scenarios, from simply running and forgetting about a process, reading from or writing to the process, to reading from and writing to it.

For the first two, popen() is very easy to use and works well. I understand that it uses some version of fork() and exec() internally, then invokes a shell to actually run the command.

For the third scenario, popen() is not an option, as it is unidirectional. Available options are:

• Manually fork() and exec(), plus pipe() and dup2() for input/output
• posix_spawn(), which internally uses the above as need be

What I noticed is that these can achieve the same that popen() does, but we can completely avoid the invoking of an additional sh. This sounds desirable, as it seems less complex.

However, I noticed that even examples on posix_spawn() that I found on the Internet do invoke a shell, so it would seem there must be a benefit to it. If it is about parsing command line arguments, wordexp() seems to do an equally good job.

What is the reason behind benefit of invoking a shell to run the desired process instead of running it directly?

---

Edit: I realized that my wording of the question didn't precisely reflect my actual interest - I was more curious about the benefits of going through sh rather than the (historical) reason, though both are obviously connected, so answers for both variations are equally relevant.

Answer 1

Invoking a shell allows you to do all the things that you can do in a shell. For example,

FILE *fp = popen("ls *", "r");

is possible with popen() (expands all files in the current directory). Compare it with:

execvp("/bin/ls", (char *[]){"/bin/ls", "*", NULL});

You can't exec ls with * as argument because exec(2) will interpret * literally.

Similarly, pipes (|), redirection (>, <, ...), etc., are possible with popen.

Otherwise, there's no reason to use popen if you don't need shell - it's unnecessary. You'll end up with an extra shell process and all the things that can go wrong in a shell go can wrong in your program (e.g., the command you pass could be incorrectly interpreted by the shell and a common security issue). popen() is designed that way. fork + exec solution is cleaner without the issues associated with a shell.

Answer 2

The glib answer is because the The POSIX standard ( http://pubs.opengroup.org/onlinepubs/9699919799/functions/popen.html ) says so. Or rather, it says that it should behave as if the command argument is passed to /bin/sh for interpretation.

So I suppose a conforming implementation could, in principle, also have some internal library function that would interpret shell commands without having to fork and exec a separate shell process. I'm not actually aware of any such implementation, and I suspect getting all the corner cases correct would be pretty tricky.