Reinstall packages automatically into virtual environment after Python major version upgrade

Question

I've got several virtual environments (dozens) lying on my disk made by the venv module of Python 3.6. Now I've upgraded to Ubuntu 19.10 in a haste and only afterwards noticed that 3.6 is not available at all for Ubuntu 19.10 from the generally acknowledged sources. I've managed to upgrade the Python versions of these virtual environments by locating bin/python3 under my home directory and running python3.7 -mvenv --upgrade on the containing folders.

Now, while python3.7 -mvenv --upgrade upgrades the Python in the virtual environment, it does nothing to reinstall my previous package versions in the lib/python3.7/site-packages under that venv. I guess I could have done this by installing Python 3.6, pip freezeing the requirements from the venv and then upgrading the venv to Python 3.7, pip install -ring - if only there was Python 3.6 install available for my new OS.

Is there any other way to do this in a rather automated fashion (perhaps mainly the pip freeze using the old lib/python3.6 directory) without me having to install Python 3.6 from source, using conda or installing 3.6 from some random PPAs? I want to upgrade all environments en masse so that in the future when I need to do something with a random environment it would continue working with Python 3.7.

Answer 1

In your new 3.7 venv you should have pkg_resources available - setuptools is automatically installed when created. If not, just pip install setuptools.

setuptools library code is actually what pip is vendoring to make pip freeze work. But you can just freeze it manually.

# in 3.7 runtime...
import pkg_resources
old_site_dir = ".venv/lib/python3.6/site-packages/"
working_set = pkg_resources.WorkingSet([old_site_dir])
for dist in working_set:
    print(dist.as_requirement())

You can throw that output in a requirements.txt file and likely have a working reconstructed site, no python3.6 runtime required.

Note that this method may not be 100% foolproof, because it is possible for projects to declare separate dependency trees for python3.6 and python3.7 by using environment markers in their distribution metadata (see PEP 508). It is also possible that items installed in your 3.6 site do not support 3.7 at all. However it is pretty uncommon to see that in a minor version bump between 3.6 and 3.7, so just using the working set should be "good enough" in practice.