Capturing standard out from a Paramiko command

Question

I have a wrapper around Paramiko's SSHClient.exec_command(). I'd like to capture standard out. Here's a shortened version of my function:

def __execute(self, args, sudo=False, capture_stdout=True, plumb_stderr=True,
  ignore_returncode=False):

  argstr = ' '.join(pipes.quote(arg) for arg in args)

  channel = ssh.get_transport().open_session()
  channel.exec_command(argstr)

  channel.shutdown_write()

  # Handle stdout and stderr until the command terminates
  captured = []

  def do_capture():
    while channel.recv_ready():
      o = channel.recv(1024)
      if capture_stdout:
        captured.append(o)
      else:
        sys.stdout.write(o)
        sys.stdout.flush()

    while plumb_stderr and channel.recv_stderr_ready():
      sys.stderr.write(channel.recv_stderr(1024))
      sys.stderr.flush()

  while not channel.exit_status_ready():
    do_capture()

  # We get data after the exit status is available, why?
  for i in xrange(100):
    do_capture()

  rc = channel.recv_exit_status()
  if not ignore_returncode and rc != 0:
    raise Exception('Got return code %d executing %s' % (rc, args))

  if capture_stdout:
    return ''.join(captured)

paramiko.SSHClient.execute = __execute

In do_capture(), whenever channel.recv_ready() tells me that I can receive data from the command's stdout, I call channel.recv(1024) and append the data to my buffer. I stop when the command's exit status is available.

However, it seems like more stdout data comes at some point after the exit status.

# We get data after the exit status is available, why?
for i in xrange(100):
  do_capture()

I can't just call do_capture() once, as it seems like channel.recv_ready() will return False for a few milliseconds, and then True, and more data is received, and then False again.

I'm using Python 2.7.6 with Paramiko 1.15.2.

Answer 1

I encountered the same problem. The problem is that after the command exited there may still be data on the stout or stderr buffers, still on its way over the network, or whatever else. I read through paramiko's source code and apparently all data's been read once chan.recv() returns empty string.

So this is my attempt to solve it, until now it's been working.

def run_cmd(ssh, cmd, stdin=None, timeout=-1, recv_win_size=1024):
    '''
    Run command on server, optionally sending data to its stdin

    Arguments:
        ssh           -- An instance of paramiko.SSHClient connected
                         to the server the commands are to be executed on
        cmd           -- The command to run on the remote server
        stdin         -- String to write to command's standard input
        timeout       -- Timeout for command completion in seconds.
                         Set to None to make the execution blocking.
        recv_win_size -- Size of chunks the output is read in

    Returns:
        A tuple containing (exit_status, stdout, stderr)
    '''

    with closing(ssh.get_transport().open_session()) as chan:
        chan.settimeout(timeout)
        chan.exec_command(cmd)
        if stdin:
            chan.sendall(stdin)
            chan.shutdown_write()

        stdout, stderr = [], []

        # Until the command exits, read from its stdout and stderr
        while not chan.exit_status_ready():
            if chan.recv_ready():
                stdout.append(chan.recv(recv_win_size))
            if chan.recv_stderr_ready():
                stderr.append(chan.recv_stderr(recv_win_size))

        # Command has finished, read exit status
        exit_status = chan.recv_exit_status()

        # Ensure we gobble up all remaining data
        while True:
            try:
                sout_recvd = chan.recv(recv_win_size)
                if not sout_recvd and not chan.recv_ready():
                    break
                else:
                    stdout.append(sout_recvd)
            except socket.timeout:
                continue

        while True:
            try:
                serr_recvd = chan.recv_stderr(recv_win_size)
                if not serr_recvd and not chan.recv_stderr_ready():
                    break
                else:
                    stderr.append(serr_recvd)
            except socket.timeout:
                continue

    stdout = ''.join(stdout)
    stderr = ''.join(stderr)

    return (exit_status, stdout, stderr)

Answer 2

I encountered the same issue.

This link (Paramiko: how to ensure data is received between commands) gave me some help, in explaining that after you get exit_status_ready() you still have to receive possible additional data. In my tests (with a couple of screens of output), in every single run, there will be additional data to read after exit_status_ready() returns True.

But the way it reads the remaining data it is not correct: it uses recv_ready() to check if there is something to read, and once recv_ready() returns False it exits. Now, it will work most of the time. But the following situation can happen: recv_ready() can return False to indicate that at that moment there is nothing to receive, but it doesn't mean that it is the end of the all data. In my tests, I would leave the test running, and sometimes it would take half an hour for the issue to appear.

I found the solution by reading the following sentence in the Channel.recv() documentation: "If a string of length zero is returned, the channel stream has closed."

So we just can have a single loop and read all the data until recv() returns zero length result. At that point channel stream is closed, but just to make sure that exit status is ready we can make additional loop and sleep until channel.exit_status_ready() returns True.

Note that this will work only with a channel without pty enabled (which is default).