update a virtualenv to match requirements.txt

Question

Currently, when our project's requirements.txt is updated, we re-build our virtualenv from scratch to ensure consistent results. The problem is that our requirements.txt is quite lengthy, and updates to it generally only touch a single package.

Is there any system that compares a virtualenv to a requirements.txt and does the minimal amount of changes to make them match?

I could possibly write this myself if I could get pip to tell me which versions it would install given a requirements.txt, but I don't see such an option.

Answer 1

Updating by pip install

I wonder, why simple

$ pip install -r requirements.txt --upgrade

would not be enough?

It does compare current virtualenv with your requirements and does only the updates, which are necessary.

Removing unwanted packages

Regarding removing unwanted packages - is that really necessary? If you really insist on it, I would create unwanted-requirements.txt and before updating packages by previous call I would perform:

$ pip uninstall -r unwanted-requirements.txt

followed by $ pip install ...`

If you keep your system as consistent as you think, this shall be working quite well and in case you would make a mistake in unwanted-requirements.txt, your system is quite likely to survive as unused package is in most situations not harmful and having removed wanted package shall recover by subsequent $ pip install ...

Speeding up installations

If your real concern is not only having consistent installations, but also have them updated quickly, than there are well working methods how to speed things up. See my SO answer https://stackoverflow.com/a/18520729/346478

Answer 2

We are doing it like that: when a commit is issued (we use git) it triggers a hook which executes a custom written bash script which activates virtualenv, runs pip install -r requirements.txt which checks that everything is up to date and installs only the packages that need upgrading and deactivates virtualenv.

I don't know if that is gonna work for you because you didn't post the details about your environment, but you get the idea. Also it doesn't matter how long is your requirements.txt because pip only reinstalls some of the packages compared to installing everything from scratch if you rebuild your virtualenv completely.

Edit:

If you also need to uninstall packages that are not in requirements.txt, you can use something like that:

pip freeze | grep -v -f requirements.txt - | xargs pip uninstall -y

And only after that:

pip install -r requirements.txt