Ubuntu + virtualenv = a mess? virtualenv hates dist-packages, wants site-packages

Question

Can someone please explain to me what is going on with python in ubuntu 9.04?

I'm trying to spin up virtualenv, and the --no-site-packages flag seems to do nothing with ubuntu. I installed virtualenv 1.3.3 with easy_install (which I've upgraded to setuptools 0.6c9) and everything seems to be installed to /usr/local/lib/python2.6/dist-packages

I assume that when installing a package using apt-get, it's placed in /usr/lib/python2.6/dist-packages/ ?

The issue is, there is a /usr/local/lib/python2.6/site-packages as well that just sits there being empty. It would seem (by looking at the path in a virtualenv) that this is the folder virtualenv uses as backup. Thus even thought I omit --no-site-packages, I cant access my local systems packages from any of my virtualenv's.

So my questions are:

1. How do I get virtualenv to point to one of the dist-packages?
2. Which dist-packages should I point it to? /usr/lib/python2.6/dist-packages or /usr/local/lib/python2.6/dist-packages/
3. What is the point of /usr/lib/python2.6/site-packages? There's nothing in there!
4. Is it first come first serve on the path? If I have a newer version of package XYZ installed in /usr/local/lib/python2.6/dist-packages/ and and older one (from ubuntu repos/apt-get) in /usr/lib/python2.6/dist-packages, which one gets imported when I import xyz? I'm assuming this is based on the path list, yes?
5. Why the hell is this so confusing? Is there something I'm missing here?
6. Where is it defined that easy_install should install to /usr/local/lib/python2.6/dist-packages?
7. Will this affect pip as well?

Thanks to anyone who can clear this up!

Answer 1

I believe Mike Orr's answer from the virtual-env mailing list seems to be the best. Note the OP published this question in both places.

Original content of mail:

Years ago Debian created /usr/local/lib/pythonVERSION/site-packages, and compiled the Python binary to include it in the default search path. Ubuntu followed Debian's lead as it normally does. The Python developers did not like this because you'd get interference with a locally-installed /usr/local/bin/python using the same site-packages directory. Ubuntu finally decided to abandon site-packages and use dist-packages instead, a name which they invented so it wouldn't interfere with anything. The loing story is out there somewhere if you google it, somewhere in the Python bug tracker or distutils SIG or such.

The system works, at least if you use the Ubuntu virtualenv package. Some people have had problems using a locally-installed virtualenv on Ubuntu because the magic sys.path entries weren't being added or something. I'm not sure about --no-site-packages because I never use that option: I run PIL and mysqldb from the Ubuntu packages because it can be hard to compile their C dependencies sometimes. (Need the right header files, Python is ignoring the header files, etc.)

So Ubuntu Python packages go into /usr/lib/pythonVERSION/dist-packages. Or that python-support directory for some reason. Locally-installed Python packages go into /usr/local/lib/pythonVERSION/dist-packages by default. Whenever I install an Ubuntu 9.04 system I run:

$ sudo apt-get install python-setuptools (6.0c9) $ sudo apt-get install python-virtualenv (1.3.3) $ sudo easy_install pip $ sudo pip install virtualenvwrapper

The virtualenvs work fine this way, although I haven't tried --no-site-packages.

I'm trying to spin up virtualenv, and the --no-site-packages flag seems to do nothing with ubuntu. I installed virtualenv 1.3.3 with easy_install (which I've upgraded to setuptools 0.6c9)

These versions are both in Ubuntu 9.04, so you're making it harder on yourself by installing them locally.

and everything seems to be installed to /usr/local/lib/python2.6/dist-packages

Yes

I assume that when installing a package using apt-get, it's placed in / usr/lib/python2.6/dist-packages/ ?

Yes

4. Is it first come first serve on the path? If I have a newer version of package XYZ installed in /usr/local/lib/python2.6/dist- packages/ and and older one (from ubuntu repos/apt-get) in /usr/lib/ python2.6/dist-packages, which one gets imported when I import xyz? I'm assuming this is based on the path list, yes?

sys.path is scanned in order. The only funny thing is that .pth eggs get put earlier or later in the path than some people expect. But if you're using pip for everything it can do (i.e. except to install pip itself, precompiled eggs, and a snapshot of a local directory that's a copy rather than an egg link), you won't have many .pth eggs anyway.

5. Why the hell is this so confusing? Is there something I'm missing here?

It's not well documented. I figured it out by scanning the web.

7. Will this affect pip as well?

Yes, pip will automatically install to /usr/local/lib/pythonVERSION/site-packages. Use "pip install -E $VIRTUAL_ENV packagename" to install into a virtualenv.