rsync over ssh - using channel created by Paramiko in Python

Question

I would like to get files from a remote machine using rsync and SSH (from within a Python program).

How do you start a local instance of rsync and introduce it to the SSH channel I have opened with Paramiko?

Answer 1

This is an old question, but still the first hit in Google when searching for "rsync over paramiko", and the only up voted item here is a comment that is unrelated to the OP's question (the link in that comment points to using ControlMaster which is not supported by Paramiko).

There is an example of how to setup local port forwarding in one of the Paramiko demos here. There is also an easier to use version of it in a pull request. You can use the local port forwarding in two ways:

1. Forward an ad-hoc local port to the remote ssh server and rsync over local port using ssh protocol. This is equivalent to running ssh -L12345:localhost:22 remote-host followed by rsync --rsh 'ssh -p 12345' sourcedir/file localhost:/destdir/file.
2. Forward an ad-hoc local port to an ad-hoc remote rsync daemon and rsync over local port using rsync protocol. This is similar to running ssh -L12345:localhost:12345 followed by rsync sourcedir/file rsync://localhost:12345/module/destdir/file, except that you need to setup an ad-hoc rsync daemon running on 12345 with a module name that points to the destdir on the remote host.

I personally prefer the 2nd approach above, though it is slightly more complex because, it skips the local ssh client and also uses rsync protocol which I think is more efficient than using ssh.

Using the ForwardServer from the above pull request, the code would look somewhat like this (depends on Fabric):

RSYNC_SPEC = """
port=12345
use chroot=false

[homedir]
log file=/tmp/rsync-ad-hoc.log
max verbosity=4
path=/home/{user}/
read only=false
"""

@task
def rsync(local_path, remote_path):
    """
    local_path: Absolute path to a local source
    remote_path: Relative path (from home directory) to a remote destination
    """
    with ForwardServer(0, "localhost", rsync_port, connections[env.host_string].get_transport()) as serv:
        local_port = serv.socket.getsockname()[1]
        run("killall rsync; rm -f /tmp/rsync-ad-hoc.log /tmp/rsync-ad-hoc.conf; :")
        put(local_path=StringIO(RSYNC_SPEC.format(user=env.user)), remote_path="/tmp/rsync-ad-hoc.conf", )
        run("rsync --daemon --config /tmp/rsync-ad-hoc.conf")
        remote_rsync_path = os.path.join("rsync://localhost:%s/homedir" % local_port, remote_path)
        # Rsync expects the root of the destination to exist.
        run("mkdir -p /home/{user}/{path}".format(user=env.user, path=remote_path))
        logging.info("Attempting rsync from (localhost, %s, %s) to (%s, %s, %s)", local_port, local_path, env.host_string, rsync_port, remote_path)
        local("rsync -avzPh --delete %s/ %s/" % (local_path, remote_rsync_path))

You can also have the function take a remote absolute path and generate the directory for the module (instead of assuming that it is relative to the user's home directory).