STDERR redirection to STDOUT lost if backticks can't exec

Question

I’m finding STDERR redirection within a backticks call can be lost if the command fails to execute. I’m stumped by the behavior I am seeing.

$ perl -e 'use strict; use warnings; my $out=`DNE`; print $out'  
Can't exec "DNE": No such file or directory at -e line 1.
Use of uninitialized value in print at -e line 1.

$ perl -e 'use strict; use warnings; my $out=`DNE 2>&1`; print $out'
Use of uninitialized value in print at -e line 1.

$ perl -e 'use strict; use warnings; my $out=`echo 123; DNE 2>&1`; print $out'
123
sh: DNE: command not found

Is my syntax incorrect?

I'm using Perl 5.8.5 on Linux.

Answer 1

Your syntax is correct, but in one case perl is dropping the error message.

In general, consider testing during initialization that your system has the command you want, and fail early if it is missing.

my $foopath = "/usr/bin/foo";
die "$0: $foopath is not executable" unless -x $foopath;

# later ...

my $output = `$foopath 2>&1`;
die "$0: $foopath exited $?" if $?;

To fully understand the differences in output, it is necessary to understand details of Unix system programming. Read on.

Unix file descriptors

Consider a simple perl invocation.

perl -e 'print "hi\n"; warn "bye\n"'

Its output is

hi
bye

Note that the output of print goes to STDOUT, the standard output, and warn writes to STDERR, the standard error. When run from a terminal, both appear on the terminal, but we can send them to different places. For example

$ perl -e 'print "hi\n"; warn "bye\n"' >/dev/null
bye

The null device or /dev/null discards any output sent to it, and so in the command above, “hi” disappears. The command above is shorthand for

$ perl -e 'print "hi\n"; warn "bye\n"' 1>/dev/null
bye

That is, 1 is the file descriptor for STDOUT. To throw away “bye” instead, run

$ perl -e 'print "hi\n"; warn "bye\n"' 2>/dev/null
hi

As you can see, 2 is the file descriptor for STDERR. (For completeness, the file descriptor for STDIN is 0.)

In the Bourne shell and its derivatives, we can also merge STDOUT and STDERR with 2>&1. Read it as “make file descriptor 2’s output go to the same place as file descriptor 1’s.”

$ perl -e 'print "hi\n"; warn "bye\n"' 2>&1
hi
bye

Terminal output does not highlight the distinction, but an extra redirection shows what’s happening. We can discard both by running

$ perl -e 'print "hi\n"; warn "bye\n"' >/dev/null 2>&1

Order matters with this family of shells that processes redirections in left-to-right order, so transposing the two yields

$ perl -e 'print "hi\n"; warn "bye\n"' 2>&1 >/dev/null
bye

This may be surprising at first. The shell first processes 2>&1 which means send STDERR to the same destination as STDOUT—which it already is: the terminal! Then it processes >/dev/null and redirects STDOUT to the null device.

This duplication of file descriptors happens by calling dup2, which is usually a wrapper for fcntl.

Redirection and pipelines

Now say we want to add a prefix to each line of the output of our command.

$ perl -e 'print "hi\n"; warn "bye\n"' | sed -e 's/^/got: /'
bye
got: hi

The order is different, but remember that STDERR and STDOUT are different streams. Notice also that only “hi” got a prefix. To get both lines, they both have to appear on STDOUT.

$ perl -e 'print "hi\n"; warn "bye\n"' 2>&1 | sed -e 's/^/got: /'
got: bye
got: hi

To construct a pipeline, the shell creates child processes with fork, performs redirections with dup2, and starts each stage of the pipeline with calls to exec in the appropriate processes. For the pipeline above, the process is similar to

• shell: fork a process to run sed
• shell: wait for exit status from sed with waitpid
  • sed: create a pipe to feed input to perl
  • sed: fork a process to run perl
  • sed: dup2 to make STDIN read from the read end of the pipe
  • sed: exec the sed command
  • sed: wait for input on STDIN
    • perl: dup2 to send STDOUT to the write end of the pipe from step 3
    • perl: dup2 to send STDERR to STDOUT’s destination
    • perl: exec the perl command
    • perl: write output and eventually exit
  • sed: receive and edit the input stream
  • sed: detect end-of-file on pipe
  • sed: reap perl’s exit status with waitpid
  • sed: exit
• shell: populate $? with the return value from waitpid

Note that the child processes are created in right-to-left order. This is because shells in the Bourne family define the exit status of a pipeline to be the exit status of the last process.

Do-It-Yourself Pipelines

You can build the above pipeline in Perl with the code below.

#! /usr/bin/env perl

use strict;
use warnings;

my $pid = open my $fh, "-|";
die "$0: fork: $!" unless defined $pid;

if ($pid) {
  while (<$fh>) {
    s/^/got: /;
    print;
  }
}
else {
  open STDERR, ">&=", \*STDOUT or print "$0: dup: $!";
  exec "perl", "-e", q[print "hi\n"; warn "bye\n"]
    or die "$0: exec: $!";
}

The first call to open does a lot of work for us, as noted in the perlfunc documentation on open:

For three or more arguments if MODE is "|-", the filename is interpreted as a command to which output is to be piped, and if MODE is "-|", the filename is interpreted as a command that pipes output to us. In the two-argument (and one-argument) form, one should replace dash ("-") with the command. See Using open for IPC in perlipc for more examples of this.

Its output is

$ ./simple-pipeline
got: bye
got: hi

The code above hardcodes the duplication of STDOUT, which we can see below.

$ ./simple-pipeline >/dev/null

Perl backticks

To capture the output of another command, perl sets up the same machinery, which you can see in pp_backtick (in pp_sys.c), which calls Perl_my_popen (in util.c) to create a child process and set up the plumbing (fork, pipe, dup2). The child does some plumbing and calls Perl_do_exec3 (in doio.c) to start the command whose output we want. There we notice a relevant comment:

/* handle the 2>&1 construct at the end */

The implementation recognizes the sequence 2>&1, duplicates STDOUT, and removes the redirection from the command to be passed to the shell.

if (*s == '>' && s[1] == '&' && s[2] == '1'
    && s > cmd + 1 && s[-1] == '2' && isSPACE(s[-2])
    && (!s[3] || isSPACE(s[3])))
{
    const char *t = s + 3;

    while (*t && isSPACE(*t))
        ++t;
    if (!*t && (PerlLIO_dup2(1,2) != -1)) {
        s[-2] = '\0';
        break;
    }
}

Later we see

PerlProc_execl(PL_sh_path, "sh", "-c", cmd, (char *)NULL);
PERL_FPU_POST_EXEC
S_exec_failed(aTHX_ PL_sh_path, fd, do_report);

Inside S_exec_failed, we find

if (ckWARN(WARN_EXEC))
    Perl_warner(aTHX_ packWARN(WARN_EXEC), "Can't exec \"%s\": %s",
                cmd, Strerror(e));

That is one of the warnings you asked about in your question.

Timelines

Let’s walk through the details of how perl processes the commands from your question.

As expected

$ perl -e 'use strict; use warnings; my $out=`DNE`; print $out'
Can't exec "DNE": No such file or directory at -e line 1.
Use of uninitialized value in print at -e line 1.

No surprises here.

A subtle detail is important to understand. The code above that handles 2>&1 in-house runs only when a condition is true of the command to be executed:

if (*s != ' ' && !isALPHA(*s) &&
    strchr("$&*(){}[]'\";\\|?<>~`\n",*s)) {

This is an optimization. If the command in backticks contains the above shell metacharacters, then perl has to hand it off to the shell. But if no shell metacharacters are present, perl can exec the command directly—saving the fork and shell startup costs.

The non-existent command DNE contains no shell metacharacters, so perl does all the work. The exec-category warning is generated because the command failed and you enabled the warnings pragma. The perlop documentation tells us that backticks or qx// returns undef in scalar context when the command fails, so that’s why you get the warning about printing the undefined value of $out.

Missing warning

$ perl -e 'use strict; use warnings; my $out=`DNE 2>&1`; print $out'
Use of uninitialized value in print at -e line 1.

Where did the failed exec warning go?

Remember the basic steps of creating a child process that runs another command:

1. Create a pipe for the child to send its output to the parent.
2. Call fork to create a nearly identical child process.
3. In the child, dup2 to connect STDOUT to the write end of the pipe.
4. In the child, exec to cause the newly created child to execute another program instead.
5. In the parent, read the contents of the pipe.

To capture the output of another command, perl goes through these steps. In preparation to attempt running DNE 2>&1, perl forks a child and in the child process causes STDERR to be a duplicate of the STDOUT, but there is another side effect.

if (!*t && (PerlLIO_dup2(1,2) != -1)) {
    s[-2] = '\0';
    break;
}

If 2>&1 is at the end of the command and the dup2 succeeds, then perl writes a NUL byte just before the redirection. This has the effect of removing it from the command, e.g., DNE 2>&1 becomes DNE! Now, with no shell metacharacters in the command, perl in the child process thinks to itself, ‘Self, we can exec this command directly.’

The call to exec fails because DNE does not exist. The child still emits the failed exec warning on STDERR. It doesn’t go to the terminal because of the dup2 that pointed STDERR to the same place as STDOUT: the write end of the pipe back to the parent.

The parent process detects that the child exited abnormally, and ignores the contents of the pipe because the result of failed command-execution is documented to be undef.

Different warning

$ perl -e 'use strict; use warnings; my $out=`echo 123; DNE 2>&1`; print $out'
123
sh: DNE: command not found

Here we see a different diagnostic of DNE not existing. The first shell metacharacter encountered is ;, so perl hands the command unchanged to the shell for execution. The echo completes normally, and then DNE fails in the shell, and the shell’s STDOUT and STDERR go back to the parent process. From perl’s perspective, the shell executed fine, so there is nothing to warn about.

Related note

When you enable the warnings pragma—a Very Good Practice!—this enables the exec warning category. To see the full list of these warnings, search the perldiag documentation for the string W exec.

Observe the difference.

$ perl -Mstrict -Mwarnings -e 'my $out=`DNE`; print $out'
Can't exec "DNE": No such file or directory at -e line 1.
Use of uninitialized value $out in print at -e line 1.

$ perl -Mstrict -Mwarnings -M-warnings=exec -e 'my $out=`DNE`; print $out'
Use of uninitialized value $out in print at -e line 1.

The latter invocation is equivalent to

use strict;
use warnings;
no warnings 'exec';

my $out = `DNE`;
print defined($out) ? $out : "command failed\n";

I like formatting my own error messages when something goes wrong with an exec, pipe open, and so on. This means I usually disable exec warnings, but it also means I have to be extra-careful to test return values.