How can I correctly install multiple non-package Distribute/virtualenv/pip ecosystems on Ubuntu?

Question

I am developing Python applications in Ubuntu. I want to setup a Distribute/virtualenv/pip ecosystem to manage my Python packages independently of any system Python packages (which I manage in Synaptic, or rather I let the system manage them for me).

I could just install the python-setuptools, python-virtualenv and python-pip system packages and be on my merry way, but I also want to be able to get latest/specific versions of Distribute, virtualenv and pip. There are no PPAs for these, so I'll have to install them manually.

A final complication, is that I want to be able to do this for multiple versions of Python. That is, set up one ecosystem for python2.6, another for python, another for python3, or on a 64-bit system another for chrooted 32-bit Python.

I'm guessing that the process would be something like:

• Using Python X install my own copy of Distribute to a location in my home folder
• Using indie Distribute, easy_install pip
• Using indie pip, install virtualenv
• Using indie virtualenv, create virtual environment
• Activate virtual environment, install packages
• Repeat for Python Y, Z and Q

What installation/configuration options am I looking for?

Answer 1

Based on Walker Hale IV's answer to a similar (but distinct! ;) ) question, there are two keys to doing this:

• you don't need to install distribute and pip because these are automatically included in a new virtual environment (and you presumably only want the versions that have been tested with virtualenv)
• you can use the virtualenv source code to create a new virtual environment, rather than using the system-installed version of virtualenv

So the workflow is:

• install Python version X on your system
• download virtualenv source code version Q (probably the latest)
• create new virtual environment with Python X and virtualenv Q
• your new VE is now running Python X and the latest stable versions of pip and distribute
• pip install any other packages
• easy_install any other packages that you can't pip install

Notes:

• In your new virtual environment, you could install new (or old) versions of distribute, pip or virtualenv. (I think)
• I don't use WH4's technique of create a bootstrap virtual environment. Instead, I create the new virtual environment from the virtualenv source each time.
• This technique should be usable on any operating system.
• If I were explaining this to someone new to the whole Distribute/pip/virtualenv ecosystem concept, I would explain it in a virtualenv-centric way.

I've written a bash script that does the basics in Ubuntu:

#! /bin/bash
# Script to create a python virtual environment
# independently of the system version of virtualenv
#
# Ideally this would be a cross-platform Python
# script with more validation and fewer TODOs,
# but you know how it is.

# = PARAMETERS =
# $1 is the python executable to use
# examples: python, python2.6, /path/to/python
# $2 is the new environment folder
# example: /path/to/env_folder/name
## TODO: should be just a name
## but I can't concatenate strings in bash
# $3 is a pip requirements file
# example: /path/to/req_folder/name.txt
# you must uncomment the relevant code below to use $3
## TODO: should be just a name
## but I can't concatenate strings in bash

# other parameters are hard-coded below
# and you must change them before first use

# = EXAMPLES OF USE =
# . env_create python2.5 /home/env/legacy
## creates environment "legacy" using python 2.5
# . env_create python /home/env/default
## creates environment "default" using whatever version of python is installed
# . env_create python3.2 /home/env/bleeding /home/req/testing.txt
## creates environment "bleeding" using python 3.2 and installs packages from testing.txt using pip

# = SET UP VARIABLES =
# Required version of virtualenv package
VERSION=1.6.4
# Folder to store your virtual environments
VE_FOLDER='/media/work/environments'
## TODO: not used because I can't concatenate strings in bash
# Folder to store bootstrap (source) versions of virtualenv
BOOTSTRAP_FOLDER='/media/work/environments/bootstrap'
## TODO: not used because I can't concatenate strings in bash
# Folder to store pip requirements files
REQUIREMENTS_FOLDER='/media/work/environments/requirements'
## TODO: not used because I can't concatenate strings in bash
# Base URL for downloading virtualenv source
URL_BASE=http://pypi.python.org/packages/source/v/virtualenv
# Universal environment options
ENV_OPTS='--no-site-packages --distribute'
# $1 is the python interpreter
PYTHON=$1
# $2 is the target folder of the new virtual environment
VE_TARGET=$2
# $3 is the pip requirements file
REQ_TARGET=$3

## = DOWNLOAD VIRTUALENV SOURCE =
## I work offline so I already have this downloaded
## and I leave this bit commented out
# cd $BOOTSTRAP_DIR
# curl -O $URL_BASE/virtualenv-$VERSION.tar.gz
## or use wget

# = CREATE NEW ENV USING VIRTUALENV SOURCE =
cd $BOOTSTRAP_FOLDER
tar xzf virtualenv-$VERSION.tar.gz
# Create the environment
$PYTHON virtualenv-$VERSION/virtualenv.py $ENV_OPTS $VE_TARGET
# Don't need extracted version anymore
rm -rf virtualenv-$VERSION
# Activate new environment
cd $VE_TARGET
. bin/activate

# = INSTALL A PIP REQUIREMENTS FILE =
## uncomment this if you want to automatically install a file
# pip install -r $REQ_TARGET

# = REPORT ON THE NEW ENVIRONMENT =
python --version
pip freeze
# deactivate
## uncomment this if you don't want to start in your environment immediately

Output looks something like this (with downloading switched off and deactivation switched on):

user@computer:/home/user$ . env_create python3 /media/work/environments/test
New python executable in /media/work/environments/test/bin/python3
Also creating executable in /media/work/environments/test/bin/python
Installing distribute...............done.
Installing pip...............done.
Python 3.2
distribute==0.6.19
wsgiref==0.1.2
user@computer:/media/work/environments/test$