twisted conch filetransfer

Question

I am trying to implement a very simple file transfer client in python using twisted conch. The client should simply transfer a few files to a remote ssh/sftp server in a programatic way. The function is given username, password, file list, destination server:directory and just needs to carry out the authentication and copying in a cross-platform way.

I have read some introductory material on twisted and have managed to make my own SSH client which just executes cat on the remote server. I am having a real difficult time extending this to moving the files over. I have taken a look at cftp.py and the filetransfer tests but am completely mystified by twisted.

Does anyone have any suggestions or references that can point me in the right direction? The SSH client I have constructed already is based off this one.

Answer 1

Doing an SFTP file transfer with Twisted Conch involves a couple distinct phases (well, they're distinct if you squint). Basically, first you need to get a connection set up with a channel open on it with an sftp subsystem running on it. Whew. Then you can use the methods of a FileTransferClient instance connected to that channel to perform whichever SFTP operations you want to perform.

The bare essentials of getting an SSH connection set up can be taken care of for you by APIs provided by modules in the twisted.conch.client package. Here's a function that wraps up the slight weirdness of twisted.conch.client.default.connect in a slightly less surprising interface:

from twisted.internet.defer import Deferred
from twisted.conch.scripts.cftp import ClientOptions
from twisted.conch.client.connect import connect
from twisted.conch.client.default import SSHUserAuthClient, verifyHostKey

def sftp(user, host, port):
    options = ClientOptions()
    options['host'] = host
    options['port'] = port
    conn = SFTPConnection()
    conn._sftp = Deferred()  
    auth = SSHUserAuthClient(user, options, conn)
    connect(host, port, options, verifyHostKey, auth)
    return conn._sftp

This function takes a username, hostname (or IP address), and port number and sets up an authenticated SSH connection to the server at that address using the account associated with the given username.

Actually, it does a little more than that, because the SFTP setup is a little mixed in here. For the moment though, ignore SFTPConnection and that _sftp Deferred.

ClientOptions is basically just a fancy dictionary that connect wants to be able to see what it's connecting to so it can verify the host key.

SSHUserAuthClient is the object that defines how the authentication will happen. This class knows how to try the usual things like looking at ~/.ssh and talking to a local SSH agent. If you want to change how authentication is done, this is the object to play around with. You can subclass SSHUserAuthClient and override its getPassword, getPublicKey, getPrivateKey, and/or signData methods, or you can write your own completely different class that has whatever other authentication logic you want. Take a look at the implementation to see what methods the SSH protocol implementation calls on it to get the authentication done.

So this function will set up an SSH connection and authenticate it. After that's done, the SFTPConnection instance comes into play. Notice how SSHUserAuthClient takes the SFTPConnection instance as an argument. Once authentication succeeds, it hands off control of the connection to that instance. In particular, that instance has serviceStarted called on it. Here's the full implementation of the SFTPConnection class:

class SFTPConnection(SSHConnection):
    def serviceStarted(self):
        self.openChannel(SFTPSession())

Very simple: all it does is open a new channel. The SFTPSession instance it passes in gets to interact with that new channel. Here's how I defined SFTPSession:

class SFTPSession(SSHChannel):
    name = 'session'

    def channelOpen(self, whatever):
        d = self.conn.sendRequest(
            self, 'subsystem', NS('sftp'), wantReply=True)
        d.addCallbacks(self._cbSFTP)


    def _cbSFTP(self, result):
        client = FileTransferClient()
        client.makeConnection(self)
        self.dataReceived = client.dataReceived
        self.conn._sftp.callback(client)

Like with SFTPConnection, this class has a method that gets called when the connection is ready for it. In this case, it's called when the channel is opened successfully, and the method is channelOpen.

At last, the requirements for launching the SFTP subsystem are in place. So channelOpen sends a request over the channel to launch that subsystem. It asks for a reply so it can tell when that has succeeded (or failed). It adds a callback to the Deferred it gets to hook up a FileTransferClient to itself.

The FileTransferClient instance will actually format and parse bytes that move over this channel of the connection. In other words, it is an implementation of just the SFTP protocol. It is running over the SSH protocol, which the other objects this example has created take care of. But as far as it is concerned, it receives bytes in its dataReceived method, parses them and dispatches data to callbacks, and it offers methods which accept structured Python objects, formats those objects as the right bytes, and writes them to its transport.

None of that is directly important to using it, though. However, before giving an example of how to perform SFTP actions with it, let's cover that _sftp attribute. This is my crude approach to making this newly connected FileTransferClient instance available to some other code which will actually know what to do with it. Separating the SFTP setup code from the code that actually uses the SFTP connection makes it easier to reuse the former while changing the latter.

So the Deferred I set in sftp gets fired with the FileTransferClient connected in _cbSFTP. And the caller of sftp got that Deferred returned to them, so that code can do things like this:

def transfer(client):
    d = client.makeDirectory('foobarbaz', {})
    def cbDir(ignored):
        print 'Made directory'
    d.addCallback(cbDir)   
    return d


def main():
    ...
    d = sftp(user, host, port)
    d.addCallback(transfer)

So first sftp sets up the whole connection, all the way to connecting a local FileTransferClient instance up to a byte stream which has some SSH server's SFTP subsystem on the other end, and then transfer takes that instance and uses it to make a directory, using one of the methods of FileTransferClient for performing some SFTP operation.

Here's a complete code listing that you should be able to run and to see a directory created on some SFTP server:

from sys import stdout

from twisted.python.log import startLogging, err

from twisted.internet import reactor
from twisted.internet.defer import Deferred

from twisted.conch.ssh.common import NS
from twisted.conch.scripts.cftp import ClientOptions
from twisted.conch.ssh.filetransfer import FileTransferClient
from twisted.conch.client.connect import connect
from twisted.conch.client.default import SSHUserAuthClient, verifyHostKey
from twisted.conch.ssh.connection import SSHConnection
from twisted.conch.ssh.channel import SSHChannel


class SFTPSession(SSHChannel):
    name = 'session'

    def channelOpen(self, whatever):
        d = self.conn.sendRequest(
            self, 'subsystem', NS('sftp'), wantReply=True)
        d.addCallbacks(self._cbSFTP)


    def _cbSFTP(self, result):
        client = FileTransferClient()
        client.makeConnection(self)
        self.dataReceived = client.dataReceived
        self.conn._sftp.callback(client)



class SFTPConnection(SSHConnection):
    def serviceStarted(self):
        self.openChannel(SFTPSession())


def sftp(user, host, port):
    options = ClientOptions()
    options['host'] = host
    options['port'] = port
    conn = SFTPConnection()
    conn._sftp = Deferred()
    auth = SSHUserAuthClient(user, options, conn)
    connect(host, port, options, verifyHostKey, auth)
    return conn._sftp


def transfer(client):
    d = client.makeDirectory('foobarbaz', {})
    def cbDir(ignored):
        print 'Made directory'
    d.addCallback(cbDir)
    return d


def main():
    startLogging(stdout)

    user = 'exarkun'
    host = 'localhost'
    port = 22
    d = sftp(user, host, port)
    d.addCallback(transfer)
    d.addErrback(err, "Problem with SFTP transfer")
    d.addCallback(lambda ignored: reactor.stop())
    reactor.run()


if __name__ == '__main__':
    main()

makeDirectory is a fairly simple operation. The makeDirectory method returns a Deferred that fires when the directory has been created (or if there's an error doing so). Transferring a file is a little more involved, because you have to supply the data to send, or define how received data will be interpreted if you're downloading instead of uploading.

If you read the docstrings for the methods of FileTransferClient, though, you should see how to use its other features - for actual file transfer, openFile is mainly of interest. It gives you a Deferred which fires with an ISFTPFile provider. This object has methods for reading and writing file contents.