When would the -e, --editable option be useful with pip install?

Question

As the man page says it:

-e,--editable <path/url>
     Install a project in editable mode (i.e.  setuptools "develop mode") from a local project path or a VCS url.

So you would use this when trying to install a package locally, most often in the case when you are developing it on your system. It will just link the package to the original location, basically meaning any changes to the original package would reflect directly in your environment.

Some nuggets around the same here and here.

An example run can be:

pip install -e .

or

pip install -e ~/ultimate-utils/ultimate-utils-proj-src/

note the second is the full path to where the setup.py would be at.

Concrete example of using --editable in development

If you play with this test package as in:

cd ~
git clone https://github.com/cirosantilli/vcdvcd
cd vcdvcd
git checkout 5dd4205c37ed0244ecaf443d8106fadb2f9cfbb8
python -m pip install --editable . --user

it outputs:

Obtaining file:///home/ciro/bak/git/vcdvcd
Installing collected packages: vcdvcd
  Attempting uninstall: vcdvcd
    Found existing installation: vcdvcd 1.0.6
    Can't uninstall 'vcdvcd'. No files were found to uninstall.
  Running setup.py develop for vcdvcd
Successfully installed vcdvcd-1.0.6

The Can't uninstall 'vcdvcd' is normal: it tried to uninstall any existing vcdvcd to then replace them with the "symlink-like mechanism" that is produced in the following steps, but failed because there were no previous installations.

Then it generates a file:

~/.local/lib/python3.8/site-packages/vcdvcd.egg-link

which contains:

/home/ciro/vcdvcd
.

and acts as a "symlink" to the Python interpreter.

So now, if I make any changes to the git source code under /home/ciro/vcdvcd, it reflects automatically on importers who can from any directory do:

python -c 'import vcdvcd'

Note however that at my pip version at least, binary files installed with --editable, such as the vcdcat script provided by that package via scripts= on setup.py, do not get symlinked, just copied to:

~/.local/bin/vcdcat

just like for regular installs, and therefore updates to the git repository won't directly affect them.

By comparison, a regular non --editable install from the git source:

python -m pip uninstall vcdvcd
python -m pip install --user .

produces a copy of the installed files under:

~/.local/lib/python3.8/site-packages/vcdvcd

Uninstall of an editable package as done above requires a new enough pip as mentioned at: How to uninstall editable packages with pip (installed with -e)

Tested in Python 3.8, pip 20.0.2, Ubuntu 20.04.

Recommendation: develop directly in-tree whenever possible

The editable setup is useful when you are testing your patch to a package through another project.

If however you can fully test your change in-tree, just do that instead of generating an editable install which is more complex.

E.g., the vcdvcd package above is setup in a way that you can just cd into the source and do ./vcdcat without pip installing the package itself (in general, you might need to install dependencies from requirements.txt though), and the import vcdvcd that that executable does (or possibly your own custom test) just finds the package correctly in the same directory it lives in.

From Working in "development" mode:

Although not required, it’s common to locally install your project in “editable” or “develop” mode while you’re working on it. This allows your project to be both installed and editable in project form.

Assuming you’re in the root of your project directory, then run:

pip install -e .

Although somewhat cryptic, -e is short for --editable, and . refers to the current working directory, so together, it means to install the current directory (i.e. your project) in editable mode.

Some additional insights into the internals of setuptools and distutils from “Development Mode”:

Under normal circumstances, the distutils assume that you are going to build a distribution of your project, not use it in its “raw” or “unbuilt” form. If you were to use the distutils that way, you would have to rebuild and reinstall your project every time you made a change to it during development.

Another problem that sometimes comes up with the distutils is that you may need to do development on two related projects at the same time. You may need to put both projects’ packages in the same directory to run them, but need to keep them separate for revision control purposes. How can you do this?

Setuptools allows you to deploy your projects for use in a common directory or staging area, but without copying any files. Thus, you can edit each project’s code in its checkout directory, and only need to run build commands when you change a project’s C extensions or similarly compiled files. You can even deploy a project into another project’s checkout directory, if that’s your preferred way of working (as opposed to using a common independent staging area or the site-packages directory).

To do this, use the setup.py develop command. It works very similarly to setup.py install, except that it doesn’t actually install anything. Instead, it creates a special .egg-link file in the deployment directory, that links to your project’s source code. And, if your deployment directory is Python’s site-packages directory, it will also update the easy-install.pth file to include your project’s source code, thereby making it available on sys.path for all programs using that Python installation.

It is important to note that pip uninstall can not uninstall a module that has been installed with pip install -e. So if you go down this route, be prepared for things to get very messy if you ever need to uninstall. A partial solution is to (1) reinstall, keeping a record of files created, as in sudo python3 -m setup.py install --record installed_files.txt, and then (2) manually delete all the files listed, as in e.g. sudo rm -r /usr/local/lib/python3.7/dist-packages/tdc7201-0.1a2-py3.7.egg/ (for release 0.1a2 of module tdc7201). This does not 100% clean everything up however; even after you've done it, importing the (removed!) local library may succeed, and attempting to install the same version from a remote server may fail to do anything (because it thinks your (deleted!) local version is already up to date).