twisted perspective broker server side delay

Question

I am using twisted's Perspective Broker to talk between the client and the server. The client requests from the server a remote method 'remote_ssh'. This causes the PB server to initiate an SSH connection using Paramiko's SSH library and retrieve a configuration from a remote device.

This all works fine, but when I have this execute for several remote devices I see the following behavior--the Perspective Broker client will send all of the requests to the PB server. The PB server will then execute these requests one-by-one (which is fine), but it won't return ANY of the results until they ALL complete.

Here is the relevant PB server side code:

class RMethods(pb.Root):

    def remote_ssh(self, aDict):

        self.login('SSH', aDict)        # Login to remote device
        response = self.aSSH.retrieve() # Retrieve the config
        self.aSSH.close()

        return response


if __name__ == "__main__":
    reactor.listenTCP(1885, pb.PBServerFactory(RMethods()))
    reactor.run()

From looking at various system level information (TCPDump and netstat), I see the following (assume 5 calls of the remote method):

Call remote_ssh from PB Client to PB Server for the five remote devices happens at about the same time

   self.login device 1
   self.aSSH.retrieve() device 1
   self.aSSH.close() device 1

   self.login device 2
   self.aSSH.retrieve() device 2
   self.aSSH.close() device 2

   ...

   self.login device 5
   self.aSSH.retrieve() device 5
   self.aSSH.close() device 5

   return results for all 5 devices  

I am not understanding why it waits to return the results (i.e. why does it wait until device5 completes before the results for device1 are returned).

Answer 1

This happens because the responses can't be sent until the reactor has a chance to run. If all five requests arrive at about the same time, then the reactor may wake up once and notice all five of them at the same time. It will dispatch all five of them to the PB server. The remote_ssh method will service one of them, blocking the whole time. When it is done, the response will (almost certainly) be queued. Then the remote_ssh method will service the next one, and that response will be queued. And so on until all of the requests have been handled. Then the reactor will have completed dispatching the original group of 5 events and move on to the next thing. When it moves on, it will find data ready to be sent and start to send it.

In other words, blocking prevents the reactor from operating, including preventing it from sending output that is ready to be sent.

You could try Twisted Conch as your SSH client instead, which may let you do the SSH work without blocking, or you could try using your existing SSH code with threads (assuming it can be made thread-safe) or multiple processes.