Multiple python installations and pip, dude, where is my site-packages?

Question

As we all know, Mac OS ships with its own python pre installed.

The recommendation seems to be to leave that alone and use homebrew to install a fresh python into the system.

My issue is that after installing python (and pip) using homebrew, pip is installing packages into the Mac OS site-packages instead of my own. I have confirmed I am running the "homebrew" pip:

$ which pip
/usr/local/bin/pip

But then when I pip install something I can se it is installed at:

/lib/python2.7/site-packages

Pip should be installing at /usr/local/lib/python2.7/site-packages unless i'm miss understanding something.

The surprising thing is that checking with -V yields a surprising result:

pip -V
pip 7.1.0 from /usr/local/lib/python2.7/site-packages (python 2.7)

Running pip list just after running pip install does not show the packages that were supposedly just installed by it but went to the wrong site-packages.

Adding to this, the packages installed on the /lib/python2.7/site-packages are not recognized by my $PYTHONPATH and as such I cannot use them.

To add even more confusion, I decided to use a virtualenv, but I was amazed as even using pip with the virtualenv active kept installing to the /lib/python2.7/site-packages instead of the virtualenv site-packages.

So, somehow I ended up with a homebrew pip, that installs packages outside of the homebrew site-packages and a python interpreter that cant use the packages installed by pip.

How do you recommend I go about finding the root cause and having a smooth python experience? :)

Answer 1

I think after you activate a virtualenv your python path should point to that environments site-package location--if not it's probably not activated. Only once you activate it will you run pip so it installs in that virtual env's site-packages. if it's not activated, it will go in whatever other site-packages it already knows about:

• Step 1: create a virtual env
  • a la... virtualenv venv
  • Do this only once!
• Step 2: Activate the vitual env
  • something like source /venv/bin/activate
  • Needs doing every time you want to use this virtual environment
• Step 3: run pip commands, watch them get installed in the virtual env site-packages!

If you do step 3 before step 2 your not actually using the virtual environment you created, so all bets are off--That's probably the reason pip is still installing to the old location.

Now, my overall recommendation is to go further and use pyenv to install specific version of python into your /Users/username/.pyenv folder and abandon both the default OSX and homebrew packages. It's simple and you can control easily the exact version of python to use by simple issuing of command to change versions.

THEN use virtualenv in python2 or pyvenv if in python3 (not to be confused with pyenv) to build vitual environments with their own local site-packages to store pip modules. When you activate a virtualenv, your $PYTHONPATH will switch to the specific location.

The flow would then be:

• Use pyenv to pull down and switch to a specific version of python you want to use--overriding homebrew and the OSX version.
• Create your vitrualenv. This will create a bin that will link to the pyenv python stack you just specified in the previous step.
• Activate the virtual env, and proceed.

Totally control your environment!