How does one create a Python virtualenv in Jenkins?

Question

I am using a Makefile to provide consistent single commands for setting up a virtualenv, running tests, etc. I have configured my Jenkins instance to pull from a mercurial repo and then run "make virtualenv", which does this:

virtualenv --python=/usr/bin/python2.7 --no-site-packages . && . ./bin/activate && pip install -r requirements.txt 

But for some reason it insists on using the system-installed pip and trying to install my package dependencies in the system site-packages rather than the virtualenv:

error: could not create '/usr/local/lib/python2.7/dist-packages/flask': Permission denied 

If I add some debugging commands and explicitly point to the pip in my virtualenv, things get even more confusing:

virtualenv --python=/usr/bin/python2.7 --no-site-packages . && . ./bin/activate && ls -l bin && which pip && pwd && ./bin/pip install -r requirements.txt 

Which generates the following output:

New python executable in ./bin/python2.7 Not overwriting existing python script ./bin/python (you must use ./bin/python2.7) Installing setuptools, pip...done. Running virtualenv with interpreter /usr/bin/python2.7 

It appears Jenkins doesn't rebuild the environment from scratch for each build, which strikes me as an odd choice, but shouldn't effect my immediate issue

The output from the "ls -l bin" shows pip to be installed in the virtualenv and executable:

-rw-r--r-- 1 jenkins jenkins    2248 Apr  9 21:14 activate -rw-r--r-- 1 jenkins jenkins    1304 Apr  9 21:14 activate.csh -rw-r--r-- 1 jenkins jenkins    2517 Apr  9 21:14 activate.fish -rw-r--r-- 1 jenkins jenkins    1129 Apr  9 21:14 activate_this.py -rwxr-xr-x 1 jenkins jenkins     278 Apr  9 21:14 easy_install -rwxr-xr-x 1 jenkins jenkins     278 Apr  9 21:14 easy_install-2.7 -rwxr-xr-x 1 jenkins jenkins     250 Apr  9 21:14 pip -rwxr-xr-x 1 jenkins jenkins     250 Apr  9 21:14 pip2 -rwxr-xr-x 1 jenkins jenkins     250 Apr  9 21:14 pip2.7 lrwxrwxrwx 1 jenkins jenkins       9 Apr 10 19:31 python -> python2.7 lrwxrwxrwx 1 jenkins jenkins       9 Apr 10 19:31 python2 -> python2.7 -rwxr-xr-x 1 jenkins jenkins 3349512 Apr 10 19:31 python2.7 

The output of "which pip" seems to want to use the correct one:

/var/lib/jenkins/jobs/Run Tests/workspace/bin/pip 

My current working directory is what I expect it to be:

/var/lib/jenkins/jobs/Run Tests/workspace 

But... wtf?

/bin/sh: 1: ./bin/pip: Permission denied make: *** [virtualenv] Error 126 Build step 'Execute shell' marked build as failure Finished: FAILURE 

Answer 1

I have being using python virtualenvs with Jenkins every day in the last two years, at multiple companies and for small side projects and cannot say I found "THE" answer. Still, I hope that sharing my experience would help others save time. Hopefully I will get further feedback in order to make the decision easier.

• Avoid ShiningPanda - it not well maintained, incompatible with Jenkins2 pipelines and prevents execution of jobs in parallel. Also it has the bad habit of leaving orphan environments on disk.
• DIY via bash and virtualenv is my current favourite. Create it inside $WORKSPACE and, if not always cleaning, run relocatable before activating them. This is because jenkins workspace folder disk location can change between executions of job N and N+1.

If you use multiple builders that do need the same virtualenv, the easiest way is to dump your environment to a file and source it at the beginning of the new builder.

To ease the maintenance I am planning to investigate these:

• direnvm
• virtualenv-wrapper (mkvirtualenv)
• pyenv

If you hit the shebang command line limits the best thing to do is to change your jenkins home directory to just /j.