Packaging a twistd plugin using pyinstaller

Question

I created a nice python Twisted app with a plugin for the twistd runner, as specified in the Twisted Documentation: http://twistedmatrix.com/documents/current/core/howto/tap.html. I am having problems packaging this with PyInstaller: my twistd plugin is not found during execution of the frozen application.

To ship my project, I created my own top-level startup script using the twistd runner modules, e.g.

#!/usr/bin/env python
from twisted.scripts.twistd import run
from sys import argv
argv[1:] = [
  '--pidfile', '/var/run/myapp.pid',
  '--logfile', '/var/run/myapp.log',
  'myapp_plugin'
]
run()

Next, I use PyInstaller to freeze this as a single directory deployment. Executing the frozen script above fails as it cannot find my twistd plugin (edited for brevity):

~/pyinstall/dist/bin/mystartup?16632/twisted/python/modules.py:758:
UserWarning: ~/pyinstall/dist/mystartup?16632 (for module twisted.plugins)
not in path importer cache (PEP 302 violation - check your local configuration).

~/pyinstall/dist/bin/mystartup: Unknown command: myapp_plugin

Normally, Twistd inspects the Python system path to discover my plugin in twisted/plugins/myapp_plugin.py. If I print the list of twistd plugins in my startup script, the list is empty in the executable resulting from PyInstaller, e.g.

from twisted.plugin import IPlugin, getPlugins
plugins = list(getPlugins(IPlugin))
print "Twistd plugins=%s" % plugins

I use a somewhat default PyInstaller spec file, no hidden imports or import hooks specified.

I like the functionality of twistd with logging, pid files, etc, so I would like to avoid having to abandon the twistd runner altogether to circumvent the plugin issue. Is there a way to ensure my twistd plugin is found in the frozen executable?

Answer 1

I found a workaround by reverse engineering some of the twisted code. Here I hardcode the plugin import. This works fine with PyInstaller for me.

#!/usr/bin/env python
import sys

from twisted.application import app
from twisted.scripts.twistd import runApp, ServerOptions

import myapp_plugin as myplugin


plug = myplugin.serviceMaker


class MyServerOptions(ServerOptions):
    """
    See twisted.application.app.ServerOptions.subCommands().
    Override to specify a single plugin subcommand and load the plugin
    explictly.
    """
    def subCommands(self):
        self.loadedPlugins = {plug.tapname:plug}
        yield (plug.tapname,
               None,
               # Avoid resolving the options attribute right away, in case
               # it's a property with a non-trivial getter (eg, one which
               # imports modules).
               lambda plug=plug: plug.options(),
               plug.description)

    subCommands = property(subCommands)


def run():
    """
    Replace twisted.application.app.run()
    To use our ServerOptions.
    """
    app.run(runApp, MyServerOptions)


sys.argv[1:] = [
    '--pidfile', '/var/run/myapp.pid',
    '--logfile', '/var/run/myapp.log',
    plug.tapname] + sys.argv[1:]

run()