What's the pythonic way to package a web app with a generation step?

Question

I have a Python-based web app that I'm trying to package as a setuptools package so that it can be installed using pip and/or python setup.py xxxxx. This web app also contains static files for a React front end. I use webpack (and therefore node.js) to generate the JavaScript bundle for the website. I'm trying to figure out the most pythonic way to package this. From googling around a bit, I found nodeenv which seems relevant.

Ideally, I would like this package to have the following traits:

• When installed with pip install or python setup.py install it should not install node and webpack, but the installed package should include the webpack output.

• The webpack-generated output should not need to be checked into the source repo. (i.e. it will need to be generated at some point or another in the packaging process.)

• When the package is set up for development via pip install -e or python setup.py develop, it should install node and webpack (I suspect the aforementioned nodeenv will be useful in this regard.) It should also run webpack at this time, so that afterwards, the webpack-generated content exists.

• If it were easy, it would also be cool if webpack could be started in "watch" mode when the virtualenv is activated, and stopped when it's deactivated (but this is totally a stretch goal.)

My hunch, given these requirements, is that I will need to subclass the sdist command to cause the webpack output to be generated at source distribution generation time. I'm also guessing I'll need to subclass the develop command to inject the development-only requirements.

It seems like this is a bridge that someone must have crossed before. Anyone have any pointers?

Answer 1

I think you're better off splitting these concerns into different build steps, if we disect your process a bit, these steps come up (assuming that node, npm and the virtualenv are already installed on your box)

1. Install the required python modules in the local virtualenv.
2. Install webpack and the npm modules needed to run the webpack script.
3. Run the webpack config so your static assets will be compiled locally.

Each of these steps represent a command that can end up in a Makefile or just a simple shell script for example (or use Fabric if you want to stick with python) so you would end up with the following commands:

• python-requirements
• node-requirements
• build-static
• build -> python-requirements, node-requirements, build-static

Now you can run these commands at will! If you're deploying you would run build for example, which will run each step in succession.

Answer 2

We're not the same deployment system but seek the same sort of thing: no need for node on production, but build with webpack for the final deployment. We're using docker to run up a temporal build machine...

The builder installs all the distribution packages it needs, then checks out the code, calls setup.py to build itself, runs myriad tests, and finally deploys the build dir to prod.

1. So I've left it up to the docker's config to ensure that nodejs and npm are installed by adding curl... && apt-get etc. to the Dockerfile.

2. I've subclassed the sdist and modified the run command to just run npm install and webpack on the commandline when it runs.

So in setup.py

setup(
    name='myapp',
    ...
    cmdclass={'sdist': MySdistCommand}
    ...)

Then MySdistCommand is

from setuptools.command.sdist import sdist

class MySdistCommand(sdist):
   def run(self):
       import subprocess
       subprocess.check_call(['npm', 'install'])
       subprocess.check_call(['./node_modules/.bin/webpack', '-p'])
       sdist.run(self)

Which seems to work so far. I'll let you know if quirks appear when we try to deploy it to prod (via a rather contorted docker+puppet system). I'm not sure what directory it will find itself in when it tries to run for real, but it works in dev. :-D