Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Adding shared python packages to multiple virtualenvs

Current Python Workflow

I have pip, distribute, virtualenv, and virtualenvwrapper installed into my Python 2.7 site-packages (a framework Python install on Mac OS X). In my ~/.bash_profile I have the line

export PIP_DOWNLOAD_CACHE=$HOME/.pip_download_cache

This gives a workflow as follows:

$ mkvirtualenv pip-test
$ pip install nose        # downloaded and installed from PyPi
$ pip install mock        # downloaded and installed from PyPi
$ mkvirtualenv pip-test2
$ pip install nose        # installed from pip's download cache
$ pip install mock        # installed from pip's download cache

Questions

Since I'm not downloading packages that have been previously installed in another virtualenv, this workflow saves time and bandwidth. However, it doesn't save disk space, since each package will be installed into each virtualenv. Therefore, I'm wondering:

  • Question #1 Is there a modification to this workflow that would allow me to conserve disk space by having multiple virtualenvs reference one Python package that is not installed in my Python 2.7 site-packages?

I've tried using add2virtualenv which is part of virtualenvwrapper. While this "adds the specified directories to the Python path for the currently-active virtualenv," it doesn't add any of the executables found in the virtualenv/bin directory. Therefore, the following will fail:

$ mkvirtualenv pip-test3
$ add2virtualenv ~/.virtualenvs/pip-test/lib/python2.7/site-packages/nose/
$ nosetests   # Fails since missing ~/.virtualenvs/pip-test3/bin/nosetests
  • Question #2 Am I missing something about the way add2virtualenv works?
  • Question #1 Rephrased Is there a better method than add2virtualenv that allows multiple virtualenvs to reference one Python package that is not installed in my Python 2.7 site-packages?
  • Question #3 If there is a method to install a shared Python package into multiple virtualenvs, is there a performance penalty that isn't there compared to installing Python packages separately into each virtualenv?
  • Question #4 Should I just give up on conserving disk space and stick with my current workflow?
like image 858
Matthew Rankin Avatar asked Sep 11 '10 20:09

Matthew Rankin


People also ask

Can you have multiple virtual environments in Python?

Create a virtual environment for each project. Run each application (the first time you try to do this, each application will fail). Research and install the packages that you think you might need to successfully run each application. Successfully run each application.

Can you install multiple versions of Python package?

easy_install allows simultaneous installation of different versions of the same project into a single environment shared by multiple programs which must require the appropriate version of the project at run time (using pkg_resources ).

Is there a way to install all Python modules at once using pip?

You can install from a requirement file: pip install -r requirements. txt . You can also list all installed packages and put it in a requirement file: pip freeze > requirement.

How do you create multiple virtual environments in Python?

To set one up you choose a folder somewhere, and run virtualenv on it; that will create a Python and a pip executable. Installing packages with that pip will then cause them only to be installed for that particular Python.) just create another virtualenv in other path.


1 Answers

Unless you are doing development on an embedded system, I find that chasing disk space in this way is always counter-productive. It took me a long time to reach this realization, because I grew up when a very large hard drive was a few megabytes in size, and RAM was measured in K. But today, unless you are under very special and unusual constraints, the benefit of having your projects be orthogonal (you can delete any directories on your system anywhere outside your project, and have its Python packages still there) seems to always far outweigh the disk-space benefit that, if you're busy developing, you'll never — in my experience — even notice anyway.

So I guess that's the lesson I'm offering from my own experience: you'll never notice the disk space you've lost, but you will notice it if trying to clean up a directory in one place on your disk breaks projects under development somewhere else.

like image 120
Brandon Rhodes Avatar answered Sep 28 '22 19:09

Brandon Rhodes