Non-blocking file access with Twisted

Question

I'm trying to figure out if there is a defacto pattern for file access using twisted. Lots of examples I've looked at (twisted.python.log, twisted.persisted.dirdbm, twisted.web.static) actually don't seem to worry about blocking for file access.

It seems like there should be some obvious interface, probably inheriting from abstract.FileDescriptor, that all file access should be going through it as a producer/consumer.

Have I missed something or is it just that the primary use for twisted in asynchronous programming is for networking and it hasn't really been worked out for other file descriptor operations, not worrying about the purity of non-blocking IO ?

Answer 1

I think you are looking for the fdesc module. For more information on non-blocking I/O in Python, you can also watch this video.

Answer 2

There is an open ticket for this in Twisted - #3983.

Answer 3

After plenty of searching, trial, and error, I finally figured how to use fdesc.

from __future__ import print_function

from twisted.internet.task import react
from twisted.internet import stdio, protocol
from twisted.internet.defer import Deferred
from twisted.internet.fdesc import readFromFD, setNonBlocking


class FileReader(protocol.Protocol):
    def __init__(self, filename):
        self.f = open(filename, 'rb')

    def dataReceived(self, data):
        self.transport.write(data)

    def connectionMade(self):
        fd = self.f.fileno()
        setNonBlocking(fd)
        readFromFD(fd, self.dataReceived)

    def connectionLost(self, reason):
        self.f.close()

def main(reactor, filename):
    stdio.StandardIO(FileReader(filename))

[Edit: I also just figured out a simpler way that doesn't require the use of a protocol]

def getFile(filename):
    with open(filename) as f:
        d = Deferred()
        fd = f.fileno()
        setNonBlocking(fd)
        readFromFD(fd, d.callback)
        return d


def main(reactor, filename):
    d = getFile(filename)
    return d.addCallback(print)

Run either like so:

react(main, ['/path/to/file'])

Answer 4

The fdesc module might be useful for asynchronously talking to a socket or pipe, but when given an fd that refers to an ordinary filesystem file, it does blocking io (and via a rather odd interface at that). For disk io, fdesc is effectively snake oil; don't use it.

As of May 2017, the only reasonable way to get async disk io in twisted is by wrapping synchronous io calls in a deferToThread.