automate switching between environments with different dependencies for python projects

Question

I'm working on different python projects that use different versions of the same library. When I install libraries with pip, they are installed globally, and it's a single version. The only way I know of to have different versions of the same library is to create a conda environment for every project. However, it becomes slightly annoying to switch between environments all the time when I'm working on many projects at the same time.

I'm used to npm, where every node project has its own node_modules directory which essentially creates a virtual environment without the hassles of having to explicitly create and switch environments. Is there a way to get a workflow similar to that of npm in python? Is explicitly creating a conda environment for every single project the best option in python? If so, are there any common tools or scripts to improve this workflow? (for example, is there an equivalent of package.json or npm install for python?)

Answer 1

Actually, there's pretty much exactly the same for Python: Pipenv.

Pipenv is a wrapper over virtualenv. A virtualenv is a virtual install of Python, with specific dependencies installed, which you can activate only when you need it.

You can install Pipenv from pip:

$ pip install --user pipenv

To use Pipenv, you can initialize it in an existing directory, by specifying a Python version:

~/project$ pipenv --python 3.5
Creating a virtualenv for this project…
Pipfile: ~/project/Pipfile
Using /usr/local/bin/python3.5 (3.5.6) to create virtualenv…
⠴Running virtualenv with interpreter /usr/local/bin/python3.5
Using base prefix '/usr/local'
New python executable in ~/.local/share/virtualenvs/project-FR7x7nea/bin/python3.5
Also creating executable in ~/.local/share/virtualenvs/project-FR7x7nea/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: ~/.local/share/virtualenvs/project-FR7x7nea
Creating a Pipfile for this project…

This will create a file named Pipfile, that will keep your dependencies. It looks like this:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.5"

You can install dependencies with pipenv install, much like you would do with pip:

pipenv install numpy

This will add it into your Pipfile, and freeze its version in Pipfile.lock. You should always manage your dependencies from pipenv commands, rather than manually editing those files.

Additionally, you may pass to pipenv install a requirements.txt file, which will have the same effect as with pip. The requirements.txt file is standard when working with pip; Pipenv expects one that respects this format. For instance, the following is a valid minimal requirements.txt:

docopt

It simply means that you want docopt in any version. If you need specific version for your dependencies, pip supports this as well. You pass it to pipenv install with the -r flag:

~/project$ pipenv install -r requirements.txt
Requirements file provided! Importing into Pipfile…
Pipfile.lock (c2e94e) out of date, updating to (5a67c1)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (c2e94e)!
Installing dependencies from Pipfile.lock (c2e94e)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 2/2 — 00:00:02
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

Finally, you have to activate the virtual environment wrapped by Pipenv, to have effective access to the dependencies. This is done with pipenv shell (although you can execute a single command with pipenv run <cmd>):

~/project $ pipenv shell
(project) ~/project$ python
Python 3.5.6 (default, Dec  3 2018, 12:12:20) 
[GCC 8.2.1 20180831] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>>

---

The nice thing with this, is that you can clone a random repo from Github, initialize Pipenv, and run pipenv install -r requirements.txt to have a clean environment ready to run the project (the caveat here being that downloading all the dependencies might take some time). The dependencies will only be installed in the virtual environment, keeping your other virtual environments and your host environment clean.

More insights here: https://docs.python-guide.org/dev/virtualenvs/