Perl: How to pass IPC::Open3 redirected STDOUT/STDERR fhs

Question

I'm trying to capture the output my perl code generates both from print and similar statements and external commands.

Due to design constraints I can't use solutions like Capture::Tiny. I need to forward the output to the buffer variable as soon as it is generated and I need to be able to differentiate between STDOUT and STDERR. Ideally a solution for external commands would essentially work just like system apart from being able to capture STDOUT and STDERR instead of printing them.

My code is supposed to:

1. Save the old STDOUT/STDERR file handles.
2. Create a new ones for both STDERR and STDOUT.
3. Redirect all the output to this place.
4. Print a couple of things.
5. Restore the old filehandles.
6. Do something with the captured output, e.g. print it.

However I'm unable to capture the output generated from external commands. I can't do it with IPC::Run3 nor with IPC::Open3.

#!/usr/bin/perl -CSDAL
use warnings;
use strict;
use IPC::Open3;
#use IPC::Run3;

# Save old filehandles
open(my $oldout, ">&STDOUT") or die "Can't dup STDOUT: $!";
open(my $olderr, ">&STDERR") or die "Can't dup STDERR: $!";

my $buffer = "";

close(STDOUT);
close(STDERR);

open(STDOUT, '>', \$buffer) or die "Can't redirect STDOUT: $!";
*STDERR = *STDOUT; # In this example STDOUT and STDERR are printed to the same buffer.

print "1: Test\n";
#run3 ["date"], undef, \*STDOUT, \*STDERR; # This doesn't work as expected
my $pid = open3("<&STDIN", ">&STDOUT", ">&STDERR", "date");
waitpid($pid,0); # Nor does this.

print STDERR "2: Test\n";

open(STDOUT, ">&", $oldout) or die "Can't dup \$oldout: $!";
open(STDERR, ">&", $olderr) or die "Can't dup \$olderr: $!";

print "Restored!\n";
print $buffer;

Expected result:

Restored!
1: Test
Mo 25. Mär 13:44:53 CET 2019
2: Test

Actual result:

Restored!
1: Test
2: Test

Answer 1

I don't have a solution to offer you, however I can provide some explanations as to the behavior you are seeing.

First, IPC::Open3 is not supposed to work when your filehandles are variables; see this question for more explanations.

Now, why isn't IPC::Run3 working? First, notice that if don't redirect STDERR and run

run3 ["date"], undef, \$buffer, { append_stdout => 1 };

instead of

run3 ["date"], undef, \*STDOUT;

then it works as expected. (you need to add { append_stdout => 1 } or your previous outputs to $buffer will be overwritten)

To understand what's happening, in your program, after

open(STDOUT, '>', \$buffer) or die "Can't redirect STDOUT: $!";

Add

print STDERR ref(\$buffer), "\n"
print STDERR ref(\*STDOUT), "\n"

Which will print

SCALAR
GLOB

That's exactly what IPC::Run3::run3 will do to know what to do with the "stdout" you give it (see the source: _fh_for_child_output, which is called by run3):

• if it's a scalar, then a temporary file is used (the corresponding line is $fh = $fh_cache{$what} ||= tempfile, where tempfile is a function from File::Temp.

• On the other hand, when stdout is a GLOB (or tied to IO::Handle), that filehandle is used directly (that's this line of code).

Which explains why when you call run3 with \$buffer it works, but not with \*STDOUT.

---

When redirecting STDERR as well, and calling

run3 ["date"], undef, \$buffer, \$buffer, { append_stdout => 1, append_stderr => 1 };

, things start to appear weird. I don't understand what's happening, but I'll share here what I found, and hopefully someone will make sense of it.

I modified the source of IPC::Run3 and added

open my $FP, '>', 'logs.txt' or die "Can't open: $!";

at the beginning of the sub run3. When running, I only see

Restored!
1: Test

on STDOUT (my terminal), but logs.txt contains the date (something in the lines of Mon Mar 25 17:49:44 CET 2019).

Investing a bit reveals that fileno $FP returns 1 (which, unless I mistaken, is usually STDOUT (but you closed it, so I'm no so surprised that its descriptor can be reused)), and fileno STDOUT returns 2 (this might depend on your Perl version and other opened filehandles though). What seems to be happening is that system assumes that STDOUT is the file descriptor 1 and thus prints to $FP instead of STDOUT (I'm just guessing though).

Please feel free to comment/edit if you understand what's happening.

Answer 2

I ended up with the following code:

#!/usr/bin/perl -CSDAL
use warnings;
use strict;
use IPC::Run3;
use IO::Scalar;
use Encode;
use utf8;

# Save old filehandles
open(my $oldout, ">&STDOUT") or die "Can't dup STDOUT: $!";
open(my $olderr, ">&STDERR") or die "Can't dup STDERR: $!";

open(my $FH, "+>>:utf8", undef) or die $!;
$FH->autoflush;

close(STDOUT);
close(STDERR);

open(STDOUT, '>&', $FH) or die "Can't redirect STDOUT: $!";
open(STDERR, '>&', $FH) or die "Can't redirect STDOUT: $!";

print "1: Test\n";

run3 ["/bin/date"], undef, $FH, $FH, { append_stdout => 1, append_stderr => 1 };

print STDERR "2: Test\n";

open(STDOUT, ">&", $oldout) or die "Can't dup \$oldout: $!";
open(STDERR, ">&", $olderr) or die "Can't dup \$olderr: $!";

print "Restored!\n";
seek($FH, 0, 0);
while(<$FH>)
{
  # No idea why this is even required
  print Encode::decode_utf8($_);
}
close($FH);

This is far from what I originally wanted, but appears to be working at least.

The issues I have with this are:

1. I need an anonymous file handle creating clutter on the hard disk.
2. For some reason I need to fix the encoding manually.

Thank you very much to the people who dedicated their time helping me out here.

Answer 3

Is there a reason you need to use the parent's STDOUT and STDERR? IPC::Open3 is easily capable of redirecting the child's STDOUT and STDERR to unrelated handles in the parent which you can read from.

use strict;
use warnings;
use IPC::Open3;

my $pid = open3 undef, my $outerr, undef, 'date';
my $output = do { local $/; readline $outerr };
waitpid $pid, 0;
my $exit = $? >> 8;

This will read STDOUT and STDERR together, if you want to read them separately you need to pass my $stderr = Symbol::gensym as the third argument (as shown in the IPC::Open3 docs), and use a non-blocking loop to avoid deadlocking when reading both handles. IO::Async::Process or similar can fully automate this for you, but IPC::Run3 provides a much simpler solution if you only need to store the output in scalar variables. IPC::Run3 and Capture::Tiny can also both easily be fatpacked for deployment in scripts.