Running subprocess within different virtualenv with python

Question

Let's say I have 2 different versions of my app installed in 2 different virtualenvironments. myapp v1.0 and myapp v2.0.

Now I would like to compare those. The comparison is written in python itself. What would be the best way to do that? Let's assume I can run them separately and both write an output file, which I can compare later.

One way to do that would be to write a bash script (that's what I have currently). I activate one virtualenv, run myapp v1.0, activate another virtualenv, run myapp v2.0. Later run a comparison module on those files. But I would like to add some more dynamics there (take some optional arguments etc) which would be easier with python.

Edit:

Currently I have something like that (a bash script):

source virtualenv1/bin/activate
python my_script.py
deactivate

source virtualenv2/bin/activate
python my_other_script.py
deactivate

python my_comparison_script.py

instead, I'd like to do only:

python my_comparison_script.py

and my scripts would be run inside this.

Answer 1

The accepted answer does not address the problem of 'activating' a virtualenv in a subprocess.

If you start your application with a call to the python executable, like in your example it is actually very simple: you only have to explicitly point to the executable in the virtualenv.

import subprocess

subprocess.Popen(["virtualenv1/bin/python", "my_script.py"])
subprocess.Popen(["virtualenv2/bin/python", "my_other_script.py"])

will start the processes in the respective virtualenvs.

Important

To address the concerns voiced in the comments:

If you want to run a subprocess and be sure to use the same interpreter that the current process is running in you have to use sys.executable. Also available: sys.exec_prefix to access the site-specific directory prefix where the platform-dependent Python files are installed.

If you want a much more in depth discussion of this topic, have a look at this pull request.

Answer 2

I think virtualenv documentation explains it nicely.

TL;DR

Runnig python venv binary directly is NOT the same as activating venv. You also have to change PATH and VIRTUAL_ENV variables accordingly (look at os.environ)

Source

$ source /path/to/ENV/bin/activate

This will change your $PATH so its first entry is the virtualenv’s bin/ directory. (You have to use source because it changes your shell environment in-place.) This is all it does; it’s purely a convenience.

If you directly run a script or the python interpreter from the virtualenv’s bin/ directory (e.g. path/to/ENV/bin/pip or /path/to/ENV/bin/python-script.py) then sys.path will automatically be set to use the Python libraries associated with the virtualenv. But, unlike the activation scripts, the environment variables PATH and VIRTUAL_ENV will not be modified. This means that if your Python script uses e.g. subprocess to run another Python script (e.g. via a !/usr/bin/env python shebang line) the second script may not be executed with the same Python binary as the first nor have the same libraries available to it. To avoid this happening your first script will need to modify the environment variables in the same manner as the activation scripts, before the second script is executed.

Answer 3

A simple option would be to run a series of commands with subprocess as follows (note that shell=True is risky and should only be used if you can control the input).

import subprocess
    
cmd = 'source activate my_virtualenv; python my_script.py'
subprocess.call(cmd, shell=True, executable='/bin/bash')

And repeat as needed.