Running command with Paramiko exec_command causes process to sleep before finishing

Question

I'm using Python's Paramiko module to SSH into a remote machine and Tar/ZIP a folder with a LOT of files (over 14K files and 60+gigs of data). The resulting zip is around 10 gigs itself. Now I can run the command to zip/tar directly from the machine with no problem. However, when I try to run the same command through SSHClient.exec_command, it runs for a bit, but eventually the zipping process on the remote machine goes to sleep. And the recv_exit_status just hangs indefinitely. Here is the code I'm using:

stdin, stdout, stderr = ssh.exec_command('cd myDirectory; tar -zcvf output.tgz *')
status = stdout.channel.recv_exit_status()

I also tried using Zip.

stdin, stdout, stderr = ssh.exec_command('cd myDirectory; find -name "*.gz" | zip output.zip -@')
status = stdout.channel.recv_exit_status()

In both cases, if I run the command directly from the remote machine, it finishes zipping/TARing. The result file is is like 9 gigs. But when I try it from Paramiko, it starts, goes more than half way (6ish gigs) and then the process goes to sleep!

I've monitored the processes on the remote machine using top, and the zip/tar WILL start running, but it will eventually go to sleep before finishing. And the python script will hang indefinitely.

Any ideas why this is happening?

Answer 1

It could be a timeout related. Try adding timeout param (in seconds) to the call: exec_command(timeout=20*60). This is 20 min example. See doc string from that method for more info:

def exec_command(self, command, bufsize=-1, timeout=None, get_pty=False):
    """
    Execute a command on the SSH server.  A new `.Channel` is opened and
    the requested command is executed.  The command's input and output
    streams are returned as Python ``file``-like objects representing
    stdin, stdout, and stderr.

    :param str command: the command to execute
    :param int bufsize:
        interpreted the same way as by the built-in ``file()`` function in
        Python
    :param int timeout:
        set command's channel timeout. See `Channel.settimeout`.settimeout
    :return:
        the stdin, stdout, and stderr of the executing command, as a
        3-tuple

    :raises SSHException: if the server fails to execute the command
    """

Also there is another issue that i experience which could also contribute: https://github.com/paramiko/paramiko/issues/109

Try my suggestion in https://github.com/paramiko/paramiko/issues/109#issuecomment-111621658

I also experienced this issue it is due to stdout.channel.eof_received == 0

import paramiko
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect("1.1.1.1", username="root", password="pass")
stdin, stdout, stderr = client.exec_command("service XXX start")

stdin, stdout and stderr are staying open...

>>> print stdin
<paramiko.ChannelFile from <paramiko.Channel 3 (open) window=2097152 in-buffer=50 -> <paramiko.Transport at 0x17eff90L (cipher aes128-ctr, 128 bits) (active; 1 open channel(s))>>>
>>> print stdout
<paramiko.ChannelFile from <paramiko.Channel 3 (open) window=2097152 in-buffer=50 -> <paramiko.Transport at 0x17eff90L (cipher aes128-ctr, 128 bits) (active; 1 open channel(s))>>>
>>> print stderr
<paramiko.ChannelFile from <paramiko.Channel 3 (open) window=2097152 in-buffer=50 -> <paramiko.Transport at 0x17eff90L (cipher aes128-ctr, 128 bits) (active; 1 open channel(s))>>>

So EOF was not received...

>>> print stdin.channel.eof_received
0

Usually I receive True and can just stdout.read(), but to be safe i use this workaround (which works!): Wait for a timeout, force stdout.channel.close() and then stdout.read():

>>> timeout = 30
>>> import time
>>> endtime = time.time() + timeout
>>> while not stdout.channel.eof_received:
...     sleep(1)
...     if time.time() > endtime:
...         stdout.channel.close()
...         break
>>> stdout.read()
'Starting XXX: \n[  OK  ]\rProgram started . . .\n'
>>>