Proper use of a client and Deferred with Twisted

Question

I implemented a basic SOCKS4 client with socket, but my Twisted translation isn't coming along too well. Here's my current code:

import struct
import socket

from twisted.python.failure import Failure

from twisted.internet import reactor
from twisted.internet.defer import Deferred
from twisted.internet.protocol import Protocol, ClientFactory

class Socks4Client(Protocol):
    VERSION = 4
    HOST = "0.0.0.0"
    PORT = 80

    REQUESTS = {
        "CONNECT": 1,
        "BIND": 2
    }

    RESPONSES = {
        90: "request granted",
        91: "request rejected or failed",
        92: "request rejected because SOCKS server cannot connect to identd on the client",
        93: "request rejected because the client program and identd report different user-ids"
    }

    def __init__(self):
        self.buffer = ""

    def connectionMade(self):
        self.connect(self.HOST, self.PORT)

    def dataReceived(self, data):
        self.buffer += data

        if len(self.buffer) == 8:
            self.validateResponse(self.buffer)

    def connect(self, host, port):
        data = struct.pack("!BBH", self.VERSION, self.REQUESTS["CONNECT"], port)
        data += socket.inet_aton(host)
        data += "\x00"

        self.transport.write(data)

    def validateResponse(self, data):
        version, result_code = struct.unpack("!BB", data[1:3])

        if version != 4:
            self.factory.protocolError(Exception("invalid version"))
        elif result_code == 90:
            self.factory.deferred.callback(self.responses[result_code])
        elif result_code in self.RESPONSES:
            self.factory.protocolError(Exception(self.responses[result_code]))
        else:
            self.factory.protocolError(Exception())

        self.transport.abortConnection()

class Socks4Factory(ClientFactory):
    protocol = Socks4Client

    def __init__(self, deferred):
        self.deferred = deferred

    def clientConnectionFailed(self, connector, reason):
        self.deferred.errback(reason)

    def clientConnectionLost(self, connector, reason):
        print "Connection lost:", reason

    def protocolError(self, reason):
        self.deferred.errback(reason)

def result(result):
    print "Success:", result

def error(reason):
    print "Error:", reason

if __name__ == "__main__":
    d = Deferred()
    d.addCallbacks(result, error)

    factory = Socks4Factory(d)
    reactor.connectTCP('127.0.0.1', 1080, factory)

    reactor.run()

1. I have a feeling that I'm abusing Deferred. Is this the right way to send results from my client?
2. I've read a few tutorials, looked at the documentation, and read through most of the protocols bundled with Twisted, but I still can't figure it out: what exactly is a ClientFactory for? Am I using it the right way?
3. clientConnectionLosts gets triggered a lot. Sometimes I lose the connection and get a successful response. How is that so? What does this mean, and should I treat it as an error?
4. How do I make sure that my deferred calls only one callback/errback?

Any tips are appreciated.

Answer 1

I have a feeling that I'm abusing Deferred. Is this the right way to send results from my client?

It's not ideal, but it's not exactly wrong either. Generally, you should try to keep the code that instantiates a Deferred as close as possible to the code that calls Deferred.callback or Deferred.errback on that Deferred. In this case, those pieces of code are quite far apart - the former is in __main__ while the latter is in a class created by a class created by code in __main__. This is sort of like the law of Demeter - the more steps between these two things, the more tightly coupled, inflexible, and fragile the software.

Consider giving Socks4Client a method that creates and returns this Deferred instance. Then, try using an endpoint to setup the connection so you can more easily call this method:

endpoint = TCP4StreamClientEndpoint(reactor, "127.0.0.1", 1080)
d = endpoint.connect(factory)
def connected(protocol):
    return protocol.waitForWhatever()
d.addCallback(connected)
d.addCallbacks(result, error)

One thing to note here is that using an endpoint, the clientConnectionFailed and clientConnectionLost methods of your factory won't be called. The endpoint takes over the former responsibility (not the latter though).

I've read a few tutorials, looked at the documentation, and read through most of the protocols bundled with Twisted, but I still can't figure it out: what exactly is a ClientFactory for? Am I using it the right way?

It's for just what you're doing. :) It creates protocol instances to use with connections. A factory is required because you might create connections to many servers (or many connections to one server). However, a lot of people have trouble with ClientFactory so more recently introduced Twisted APIs don't rely on it. For example, you could also do your connection setup as:

endpoint = TCP4StreamClientEndpoint(reactor, "127.0.0.1", 1080)
d = connectProtocol(endpoint, Socks4Client())
...

ClientFactory is now out of the picture.

clientConnectionLosts gets triggered a lot. Sometimes I lose the connection and get a successful response. How is that so? What does this mean, and should I treat it as an error?

Every connection must eventually be lost. You have to decide on your own whether this is an error or not. If you have finished everything you wanted to do and you called loseConnection, it is probably not an error. Consider a connection to an HTTP server. If you have sent your request and received your response, then losing the connection is probably not a big deal. But if you have only received half the response, that's a problem.

How do I make sure that my deferred calls only one callback/errback?

If you structure your code as I described in response to your first question above, it becomes easier to do this. When the code that uses callback/errback on a Deferred is spread across large parts of your program, then it becomes harder to do this correctly.

It is just a matter of proper state tracking, though. Once you give a Deferred a result, you have to arrange to know that you shouldn't give it another one. A common idiom for this is to drop the reference to the Deferred. For example, if you are saving it as the value of an attribute on a protocol instance, then set that attribute to None when you have given the Deferred its result.