Consequences for virtual env when system's Python is removed and/or updated

Question

EDIT FOR CLARITY: This question is in reference to venv, which is often confused with virtualenv.

What are the consequences of removing the system's Python installation which was the source used to create virtual environments on the system using venv?

Ultimately, what I'm looking to do is remove the 3.7 installation and install 3.8 when it is released, but I'm unsure of how strongly tethered a virtual environment is to the system environment.

Some system details that may be relevant when asking this question:

• OS = Red Hat Enterprise Linux 7
• 3.7 installed from source at Python.org, not the rh-Pythonx.x packge in the yum repo.
• Python 3.7 is installed to the default location, /usr/local/bin/python3.7
• /usr/local/bin/python3 is a link to /usr/local/bin/python3.7

Given the above details, I am under the impression that Python 3.8 would install to /usr/local/bin/python3.8 and python3 would then be pointing to that version.

Does the virtual environment have enough within it to be a stable 3.7 environment of its own? Or will it fall apart, trying to reference a system install that no longer exists?

tl;dr: What happens to all those virtual environments that were created from a Python source that is no longer existent / replaced with a newer version?

Answer 1

Bad news: You're using venv, and venv virtual environments are not completely standalone. Even when created with --copies, all it copies is the python executable itself, not the standard libraries, not even (if your install creates a shared libpython rather than a static one) libpython; it relies on the system copy. The virtual environment will break if the Python install it was based on goes away. How it breaks will differ based on how it was created. For example, if you created it with:

python3 -mvenv path/to/venv

when python3 meant Python 3.7, then replaced python3 with Python 3.8, then you can fix up the virtual env for the new version with:

python3 -mvenv --upgrade path/to/venv

but your installed third party packages will be (effectively) gone (they'll be in path/to/venv/lib/python3.7, but Python 3.8 will only be looking in path/to/venv/lib/python3.8), so you'll have to reinstall them.

If you created the virtual environment with:

python3.7 -mvenv path/to/venv

then it's completely broken (at least as documented), the --upgrade switch is only documented to work for upgrading when Python was upgraded in place; since the new Python would not be named python3.7, you can't upgrade in place. That said, --upgrade really only works well when upgrading micro versions (from 3.7.1 to 3.7.2 or the like), due to aforementioned per-minor version lib/pythonX.Y directories, so either way, you're best off just creating a new virtual environment from scratch.

To be clear, the third party virtualenv package does not have this limitation if and only if the system Python installation statically linked libpython. Oddly, while the --always-copy flag will make it copy the main binary and the standard library modules, it doesn't cause libpython itself (the interpreter core) to be copied, so if the main binary depends on a system copy of libpython.so, then removing the system copy breaks the virtual environment. If you did use --always-copy and your python executable statically linked libpython.a (ldd /path/to/python3 should show no libpython dependency), then yes, virtualenv makes much more heavyweight (in local tests on 3.6, forcing copies via the appropriate switch, freshly created venv environments were ~11 MB, while virtualenv environments were ~48 MB; sadly, my python dynamically links libpython.so, so it still wouldn't work) virtual environments that should survive the removal of the system installed copy of Python.

In any event, it's probably best to leave your Python 3.7 install in place, and just upgrade to 3.8 without removing 3.7 (are you really hard up for a few dozen MB of disk space?). Even if you replace python3 with the new 3.8 install, python3.7, libpython3.7m.so.1.0 or the like and the rest of the 3.7 standard library will continue to exist for the virtual environment to depend on; at worst you might need to manually change the symlink in the virtual environment to point to /path/to/python3.7 rather than /path/to/python3 to keep using the old version (including all your installed third party packages).

An alternative to trying to keep the old virtual environment working is to just back up the installed state of that virtual environment, delete it, install the new Python, create a fresh virtual environment, and then use the backed up state to reinstall all your packages in the upgraded virtual env. An example might be:

$ source ~/path/to/venv/bin/activate
$ pip freeze > installed_libs.txt
$ deactivate
$ rm -rf ~/path/to/venv
$ ... install new Python/remove old Python ...
$ python3 -mvenv ~/path/to/venv
$ pip install -r installed_libs.txt  # Optionally add --upgrade to install latest, not fixed versions

Answer 2

A virtualenv includes a copy of the python binary itself, so nothing should happen. See a more detailed explanation here

virtualenv solves this problem by creating a completely isolated virtual environment for each of your programs. An environment is simply a directory that contains a complete copy of everything needed to run a Python program, including a copy of the python binary itself, a copy of the entire Python standard library, a copy of the pip installer, and (crucially) a copy of the site-packages directory mentioned above. When you install a package from PyPI using the copy of pip that's created by the virtualenv tool, it will install the package into the site-packages directory inside the virtualenv directory. You can then use it in your program just as before.

Edit (accounting for venv vs virtualenv): According to the venv-documentation, it can either copy or symlink the binary:

The venv module provides support for creating lightweight “virtual environments” with their own site directories, optionally isolated from system site directories. Each virtual environment has its own Python binary (which matches the version of the binary that was used to create this environment) and can have its own independent set of installed Python packages in its site directories.