Running multiple uwsgi python versions

Question

I'm trying to deploy django with uwsgi, and I think I lack understanding of how it all works. I have uwsgi running in emperor mode, and I'm trying to get the vassals to run in their own virtualenvs, with a different python version.

The emperor configuration:

[uwsgi]
socket = /run/uwsgi/uwsgi.socket
pidfile = /run/uwsgi/uwsgi.pid
emperor = /etc/uwsgi.d
emperor-tyrant = true
master = true
autoload = true
log-date = true
logto = /var/log/uwsgi/uwsgi-emperor.log

And the vassal:

uid=django
gid=django
virtualenv=/home/django/sites/mysite/venv/bin
chdir=/home/django/sites/mysite/site
module=mysite.uwsgi:application
socket=/tmp/uwsgi_mysite.sock
master=True

I'm seeing the following error in the emperor log:

Traceback (most recent call last):
  File "./mysite/uwsgi.py", line 11, in <module>
    import site
ImportError: No module named site

The virtualenv for my site is created as a python 3.4 pyvenv. The uwsgi is the system uwsgi (python2.6). I was under the impression that the emperor could be any python version, as the vassal would be launched with its own python and environment, launched by the master process. I now think this is wrong.

What I'd like to be doing is running the uwsgi master process with the system python, but the various vassals (applications) with their own python and their own libraries. Is this possible? Or am I going to have to run multiple emperors if I want to run multiple pythons? Kinda defeats the purpose of having virtual environments.

Answer 1

The "elegant" way is building the uWSGI python support as a plugin, and having a plugin for each python version:

(from uWSGI sources)

make PROFILE=nolang

(will build a uWSGI binary without language support)

PYTHON=python2.7 ./uwsgi --build-plugin "plugins/python python27"

will build the python27_plugin.so that you can load in vassals

PYTHON=python3 ./uwsgi --build-plugin "plugins/python python3"

will build the plugin for python3 and so on.

There are various way to build uWSGI plugins, the one i am reporting is the safest one (it ensure the #ifdef are honoured).

Having said that, having a uWSGI Emperor for each python version is viable too. Remember Emperor are stackable, so you can have a generic emperor spawning one emperor (as its vassal) for each python version.

Answer 2

Pip install uWSGI

One option would be to simply install uWSGI with pip in your virtualenvs and start your services separately:

pip install uwsgi
~/.virtualenvs/venv-name/lib/pythonX.X/site-packages/uwsgi --ini path/to/ini-file

Install uWSGI from source and build python plugins

If you want a system-wide uWSGI build, you can build it from source and install plugins for multiple python versions. You'll need root privileges for this.

First you may want to install multiple system-wide python versions.

Make sure you have any dependencies installed. For pcre, on a Debian-based distribution use:

apt install libpcre3 libpcre3-dev

Download and build the latest uWSGI source into /usr/local/src, replacing X.X.X.X below with the package version (e.g. 2.0.19.1):

wget http://projects.unbit.it/downloads/uwsgi-latest.tar.gz
tar vzxf uwsgi-latest.tar.gz
cd uwsgi-X.X.X.X/
make PROFILE=nolang

Symlink the versioned folder uwsgi-X.X.X.X to give it the generic name, uwsgi:

ln -s /usr/local/src/uwsgi-X.X.X.X /usr/local/src/uwsgi

Create a symlink to the build so it's on your PATH:

ln -s /usr/local/src/uwsgi/uwsgi /usr/local/bin

Build python plugins for the versions you need:

PYTHON=pythonX.X ./uwsgi --build-plugin "plugins/python pythonXX"

For example, for python3.8:

PYTHON=python3.8 ./uwsgi --build-plugin "plugins/python python38"

Create a plugin directory in an appropriate location:

mkdir -p /usr/local/lib/uwsgi/plugins/

Symlink the created plugins to this directory. For example, for python3.8:

ln -s /usr/local/src/uwsgi/python38_plugin.so /usr/local/lib/uwsgi/plugins

Then in your uWSGI configuration (project.ini) files, specify the plugin directory and the plugin:

plugin-dir = /usr/local/lib/uwsgi/plugins
plugin = python38

Make sure to create your virtualenvs with the same python version that you created the plugin with. For example if you created python38_plugin.so with python3.8 and you have plugin = python38 in your project.ini file, then an easy way to create a virtualenv with python3.8 is with:

python3.8 -m virtualenv path/to/project/virtualenv