ssh does not reliably return output from remote command

Question

Bear with me, the description is long.

In perl I want to execute a simple remote command and capture the output using perl::Expect). However I found that my result array was sometimes empty. I've done a lot of experimenting and believe the problem is my remote ssh connection is terminating, with success, before the remote command can flush to stout.

• the behavior is the same whether running perl::Expect or ssh directly from the bash command line.
• all hosts in my data center have this behavior. The more loaded the host is the more frequent the error. For heavy hosts I fail to get a result about 15% of the time. The error rate is near 0 on lighter hosts.
• The remote command does not matter.
• I know the I'm getting past the password prompt and into the remote host. Using 'ssh -vvv' I see the remote command getting executed every time.
• I've experimented by executing a trailing dummy command and checking for the remote return code in the hopes of flushing the stdout buffer. Nothing has been reliable.

The following examples have had user/group/host names redacted. Aside from that the output is an unedited cut-and-paste.

> while true ; do echo "start"; ssh -l user host 'date; ls -l abc; echo $?'; echo "end"; done
start
Password:
Mon Oct  6 13:06:51 PDT 2014
-rw-r--r-- 1 user group 34538 Dec  6  2012 abc
0
end
start
Password:
end
start
Password:
Mon Oct  6 13:06:54 PDT 2014
end

If I tee the remote command to a dummy file then stdout is also flushed and I reliably get the output from my command every time. Unfortunately, this is a hack work around.

while true ; do echo "start"; ssh -l user host 'date | tee -a /tmp/date.txt' ; echo "end"; done

---

Don't get hung up on the perl details in this example. The interesting bit is the returned @output array. In perl I have the following command:

local::rc::Utils::executeRemoteCommand( $cmd, $pass, 0, undef, undef, \@output )

which eventually drills down to call this:

my $ssh = Expect->spawn($command);

The resulting $command is this:

ssh -l user host "ls -l abc;echo 'DONE';echo $?"

Successful 'print Dumper \@output':

$VAR1 = [
      #0
      " ",
      #1
      "-rw-r--r-- 1 user group 34538 Dec  6  2012 abc",
      #2
      "DONE",
      #3
      0
    ];

Failed 'print Dumper \@output':

$VAR1 = [
    #0
    " ",
    #1
    "DONE",
    #2
    0
];

Note how the result from the 'ls' was not captured yet the echo of 'DONE' was. I've also seen where the output array has the ls result but not the echo. I've also seen where both results were missing.

Any insight would be greatly appreciated. At this point I'm stumped and must now program these remote calls into a loop checking for an expected result. This is a terrible hack. I'd prefer to fix the root cause.

thanks much

Answer 1

I cannot reproduce it, but maybe you should try using the -t argument for ssh.