How to capture all output of a command which also requires user input from terminal?

Question

I would like to capture all output (both STDOUT and STDERR) of a command that also requires user interaction from the terminal window, i.e. it reads STDIN and then prints something to STDOUT.

Here is minimal version of the script I want to capture the output from:

user.pl:

#! /usr/bin/env perl

use feature qw(say);
use strict;
use warnings;

print "Enter URL: ";
my $ans = <STDIN>;
# do something based on $ans
say "Verification code: AIwquj2VVkwlWEBwway";
say "Access Token: bskjZO8iZotv!";

I tried using Capture::Tiny :

p.pl:

#! /usr/bin/env perl

use feature qw(say);
use strict;
use warnings;

use Capture::Tiny qw(tee_merged);

my $output = tee_merged {
    #STDOUT->autoflush(1);  # This does not work
    system "user.pl";
};

if ( $output =~ /Access Token: (.*)$/ ) {
    say $1;
}

but it does not work, since the prompt is not displayed until after the user has entered the input in the terminal.

Edit:

It seems it works fine if I replace user.pl with a python script. For example:

user.py:

#! /usr/bin/env python3

ans = input( 'Enter URL: ' )
# do something based on $ans
print( 'Verification code: AIwquj2VVkwlWEBwway' )
print( 'Access Token: bskjZO8iZotv!' )

Answer 1

TL/DR There is a solution, it's somewhat ugly, but it works. There are some minor caveats.

What's going on? The problem is actually in user.pl. The sample user.pl that you provided works like this: It starts by printing the string Enter URL: to its stdout, it then flushes its stdout and it then reads a line from its stdin. The flushing of the stdout occurs automatically by perl: when you try do read from stdin with <..> (aka readline), perl flushes stdout. It does that precisely to make programs like this behave correctly. Unfortunately, it appears that perl only implements this behavior when stdout is a tty (pseudo-terminal). If not, it does not flush stdout before reading from stdin. This is why the script works when you execute it in an interactive terminal session and it doesn't work correctly when you try to capture its output (because in that case its stdout is connected to a pipe).

How to fix this? Since user.pl misbehaves if its stdout is not a tty, we must use a tty. AFAIK, IPC::Run is the only perl module that can capture the output of a subprocess using a tty instead of a plain pipe. Unfortunately, when using a tty, IPC::Run does not allow us to redirect stdout only, it forces us to redirect stdin too. Because of that, we have to handle reading from stdin in the parent process on behalf of the child process (yikes!). Here's an example implementation of p.pl using IPC::Run:

#!/usr/bin/perl
use strict;
use warnings;
use IO::Handle;
use IPC::Run;

my $complete_output='';
my $in='';
my $out='';
my $h=IPC::Run::start ['./user.pl'],'<pty<',\$in,'>pty>',\$out;
while ($h->pumpable) {
    $h->pump;
    print $out;
    STDOUT->flush;
    if ($out eq 'Enter URL: ') {
        $in.=<STDIN>;
    }
    $complete_output.=$out;
    $out='';
}
$h->finish;
# do something with $complete_output here

So this is somewhat ugly. For example, we try do detect when the subprocess is waiting for user input (by looking for the string Enter URL:) and when it does, we read the user input in the parent process and then pass it to the child. Also notice that we have to implement the tee functionality ourselves since IPC::Run doesn't offer that.

There are some caveats. The way we handle user input, if the subprocess uses something like the readline library to support line editing, this will not work, because we do all the reading in the parent process with a simple <STDIN>. Also, because a tty is used behind the scenes instead of a pipe, all user input will be echoed to stdout. So whatever the user types in prompt, we put it in $in to send it to the process and will get it back from the process (via the $out variable). But since our terminal has also echo, the text will appear twice. One solution is filter $out to remove the user input and to prevent us from printing it.

Finally, this will not work on Windows.