Using LineReciever with Twisted Process protocols

Question

I'm trying to use twisted to handle data generated by a binary (which indefinitely dumps lines onto stdout). Since by data is inherently line-delimited, I was trying to used the LineReciever instead of trying to parse data. The following is the relevant bit of the code which seems to be causing trouble :

class ProtocolBareQDAL41xB(ProcessProtocol, LineReceiver):
    ...   
    def outReceived(self, data):
        print "Got Data:" + repr(data)
        self.dataReceived(data)

    def lineReceived(self, line):
        print "Got Line: " + line
        self._process_line(line)
    ...

This 'works' for the first of two lines in the output. I don't know yet if it works for only one line, or if it works for all but the last line. The resulting output looks something like :

$ python BareQDAL41xB.py 
Made Connection
<Process pid=16486 status=-1>
Got Data:'No device found!\nMultiple devices found! Please connect only one.\n'
Got Line: No device found!
Got Serial Number :  found!
Unhandled Error
Traceback (most recent call last):
  File "/media/ldata/code/virtualenvs/tendril/local/lib/python2.7/site-packages/twisted/python/log.py", line 101, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/media/ldata/code/virtualenvs/tendril/local/lib/python2.7/site-packages/twisted/python/log.py", line 84, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/media/ldata/code/virtualenvs/tendril/local/lib/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/media/ldata/code/virtualenvs/tendril/local/lib/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
--- <exception caught here> ---
  File "/media/ldata/code/virtualenvs/tendril/local/lib/python2.7/site-packages/twisted/internet/posixbase.py", line 597, in _doReadOrWrite
    why = selectable.doRead()
  File "/media/ldata/code/virtualenvs/tendril/local/lib/python2.7/site-packages/twisted/internet/process.py", line 274, in doRead
    return fdesc.readFromFD(self.fd, self.dataReceived)
  File "/media/ldata/code/virtualenvs/tendril/local/lib/python2.7/site-packages/twisted/internet/fdesc.py", line 94, in readFromFD
    callback(output)
  File "/media/ldata/code/virtualenvs/tendril/local/lib/python2.7/site-packages/twisted/internet/process.py", line 277, in dataReceived
    self.proc.childDataReceived(self.name, data)
  File "/media/ldata/code/virtualenvs/tendril/local/lib/python2.7/site-packages/twisted/internet/process.py", line 931, in childDataReceived
    self.proto.childDataReceived(name, data)
  File "/media/ldata/code/virtualenvs/tendril/local/lib/python2.7/site-packages/twisted/internet/protocol.py", line 604, in childDataReceived
    self.outReceived(data)
  File "BareQDAL41xB.py", line 104, in outReceived
    self.dataReceived(data)
  File "/media/ldata/code/virtualenvs/tendril/local/lib/python2.7/site-packages/twisted/protocols/basic.py", line 573, in dataReceived
    self.transport.disconnecting):
exceptions.AttributeError: 'Process' object has no attribute 'disconnecting'

processExited, status 0
processEnded, status 0

LineReciever seems to be expecting the transport to implement disconnecting.

Is it possible to use twisted's LineReciever with twisted's ProcessProtocol, or should I implement the line parser in my protocol instead?

Answer 1

LineReceiver is already a Protocol, which implements different interfaces than IProcessProtocol.

Luckily, recent versions of Twisted already contain an adapter that does what you want - which is to treat a subprocess as a stream of bytes. Rather than calling spawnProcess directly, use ProcessEndpoint, and you can pass a regular ProtocolFactory, no ProcessProtocol involved.

However, as a commenter has already pointed out, there's a bug here, where the disconnecting attribute is not formally part of ITransport, but LineReceiver (and LineOnlyReceiver) depend on it anyway, and since it's not part of the interface, ProcessEndpoint doesn't implement it. That should definitely be fixed, but in the meanwhile, we'll need to work around it.

As a happy accident, Twisted's built-in support for wrapping protocols, WrappingFactory, already has support for the disconnecting attribute, specifically because of this ugly disparity between the theory of the interface specifications and the reality of the most popular ITransport implementations. So even a do-nothing wrapper will work around the problem. You can implement this like so:

from zope.interface import implementer
from twisted.internet.interfaces import IStreamClientEndpoint
from twisted.protocols.policies import WrappingFactory

@implementer(IStreamClientEndpoint)
class DisconnectingWorkaroundEndpoint(object):
    def __init__(self, endpoint):
        self._endpoint = endpoint

    def connect(self, protocolFactory):
        return self._endpoint.connect(WrappingFactory(protocolFactory))

and then when you construct your ProcessEndpoint, do:

endpoint = DisconnectingWorkaroundEndpoint(ProcessEndpoint(...))

Sorry for the delay on answering; while you've probably worked out your own workaround, I hope this will be useful to others with the same question!