The new pip dependency resolver that was released with version 20.3 takes an inappropriately long time to install a package. On our CI pipeline yesterday, a docker build that used to take ~10 minutes timed out after 1h of pip installation messages like this (almost for every library that is installed by any dependency there is a similar log output):
INFO: pip is looking at multiple versions of setuptools to determine which version is compatible with other requirements. This could take a while.
Downloading setuptools-50.0.0-py3-none-any.whl (783 kB)
Downloading setuptools-49.6.0-py3-none-any.whl (803 kB)
Downloading setuptools-49.5.0-py3-none-any.whl (803 kB)
Downloading setuptools-49.4.0-py3-none-any.whl (803 kB)
Downloading setuptools-49.3.2-py3-none-any.whl (790 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
Downloading setuptools-49.3.1-py3-none-any.whl (790 kB)
Downloading setuptools-49.3.0-py3-none-any.whl (790 kB)
Downloading setuptools-49.2.1-py3-none-any.whl (789 kB)
Downloading setuptools-49.2.0-py3-none-any.whl (789 kB)
Downloading setuptools-49.1.3-py3-none-any.whl (789 kB)
Downloading setuptools-49.1.2-py3-none-any.whl (789 kB)
Downloading setuptools-49.1.1-py3-none-any.whl (789 kB)
Downloading setuptools-49.1.0-py3-none-any.whl (789 kB)
Downloading setuptools-49.0.1-py3-none-any.whl (789 kB)
Downloading setuptools-49.0.0-py3-none-any.whl (789 kB)
Downloading setuptools-48.0.0-py3-none-any.whl (786 kB)
Downloading setuptools-47.3.2-py3-none-any.whl (582 kB)
Downloading setuptools-47.3.1-py3-none-any.whl (582 kB)
Downloading setuptools-47.3.0-py3-none-any.whl (583 kB)
Downloading setuptools-47.2.0-py3-none-any.whl (583 kB)
Downloading setuptools-47.1.1-py3-none-any.whl (583 kB)
Downloading setuptools-47.1.0-py3-none-any.whl (583 kB)
Downloading setuptools-47.0.0-py3-none-any.whl (583 kB)
Downloading setuptools-46.4.0-py3-none-any.whl (583 kB)
Downloading setuptools-46.3.1-py3-none-any.whl (582 kB)
Downloading setuptools-46.3.0-py3-none-any.whl (582 kB)
Downloading setuptools-46.2.0-py3-none-any.whl (582 kB)
Downloading setuptools-46.1.3-py3-none-any.whl (582 kB)
Downloading setuptools-46.1.2-py3-none-any.whl (582 kB)
Downloading setuptools-46.1.1-py3-none-any.whl (582 kB)
Downloading setuptools-46.1.0-py3-none-any.whl (582 kB)
Downloading setuptools-46.0.0-py3-none-any.whl (582 kB)
Downloading setuptools-45.3.0-py3-none-any.whl (585 kB)
Downloading setuptools-45.2.0-py3-none-any.whl (584 kB)
Downloading setuptools-45.1.0-py3-none-any.whl (583 kB)
Downloading setuptools-45.0.0-py2.py3-none-any.whl (583 kB)
Downloading setuptools-44.1.1-py2.py3-none-any.whl (583 kB)
Downloading setuptools-44.1.0-py2.py3-none-any.whl (583 kB)
Downloading setuptools-44.0.0-py2.py3-none-any.whl (583 kB)
Downloading setuptools-43.0.0-py2.py3-none-any.whl (583 kB)
Downloading setuptools-42.0.2-py2.py3-none-any.whl (583 kB)
Downloading setuptools-42.0.1-py2.py3-none-any.whl (582 kB)
Downloading setuptools-42.0.0-py2.py3-none-any.whl (582 kB)
Downloading setuptools-41.6.0-py2.py3-none-any.whl (582 kB)
Downloading setuptools-41.5.1-py2.py3-none-any.whl (581 kB)
Downloading setuptools-41.5.0-py2.py3-none-any.whl (581 kB)
Downloading setuptools-41.4.0-py2.py3-none-any.whl (580 kB)
Downloading setuptools-41.3.0-py2.py3-none-any.whl (580 kB)
Downloading setuptools-41.2.0-py2.py3-none-any.whl (576 kB)
Downloading setuptools-41.1.0-py2.py3-none-any.whl (576 kB)
Downloading setuptools-41.0.1-py2.py3-none-any.whl (575 kB)
Downloading setuptools-41.0.0-py2.py3-none-any.whl (575 kB)
Downloading setuptools-40.9.0-py2.py3-none-any.whl (575 kB)
Downloading setuptools-40.8.0-py2.py3-none-any.whl (575 kB)
Downloading setuptools-40.7.3-py2.py3-none-any.whl (574 kB)
Downloading setuptools-40.7.2-py2.py3-none-any.whl (574 kB)
Downloading setuptools-40.7.1-py2.py3-none-any.whl (574 kB)
Downloading setuptools-40.7.0-py2.py3-none-any.whl (573 kB)
Downloading setuptools-40.6.3-py2.py3-none-any.whl (573 kB)
Downloading setuptools-40.6.2-py2.py3-none-any.whl (573 kB)
Downloading setuptools-40.6.1-py2.py3-none-any.whl (573 kB)
Downloading setuptools-40.6.0-py2.py3-none-any.whl (573 kB)
Downloading setuptools-40.5.0-py2.py3-none-any.whl (569 kB)
Downloading setuptools-40.4.3-py2.py3-none-any.whl (569 kB)
Downloading setuptools-40.4.2-py2.py3-none-any.whl (569 kB)
Downloading setuptools-40.4.1-py2.py3-none-any.whl (569 kB)
Downloading setuptools-40.4.0-py2.py3-none-any.whl (568 kB)
Downloading setuptools-40.3.0-py2.py3-none-any.whl (568 kB)
I am quite confused whether we are using the new pip resolver correctly, especially since
- Substantial improvements in new resolver for performance, output and error messages, avoiding infinite loops, and support for constraints files.
The behavior seen is described as backtracking in the release notes. I understand why it is there.
It specifies that I can use a constraint file (looks like a requirements.txt) that fixes the version of the dependencies to reduce the runtime using
pip install -c constraints.txt setup.py
.
What is the best way to produce this constraints file? Currently, the best way I can think of is running pip install setup.py
locally in a new virtual environment, then using pip freeze > constraints.txt
. However, this still takes a lot of time for the local install (it's been stuck for about 10 minutes now).
The notes do mention that This means the “work” is done once during development process, and so will save users this work during deployment.
With the old dependency resolver, I was able to install this package in less than a minute locally.
What is the recommended process here?
Edit: I just found out that some of the dependencies are pointing directly to out internal gitlab server. If I instead install directly from our internal package registry, it works in a couple of minutes again.
Backtracking reduces the risk that installing a new package will accidentally break an existing installed package, and so reduces the risk that your environment gets messed up. To do this, pip has to do more work, to find out which version of a package is a good candidate to install.
Upgrade Or Downgrade To Specific Pip Version If you want to upgrade or downgrade your version of pip to a specific version on a Mac, you can do this by adding a pip==<version> flag to the end of your update command.
Updating Pip When an update for pip is available, and you run a pip command, you will see a message that says, “You are using pip version xy. a, however version xy. b is available.” You can run “pip install --upgrade pip” to install and use the new version of pip.
pip 22.2.2. The PyPA recommended tool for installing Python packages.
There seems to be major update in pip
just few days old (version 22.0
, release notes + relevant issue on github).
I haven't tested it in more detail but it really seems to me that they optimized installation order calculation in complex case in such way that it resolves many issues we all encountered earlier. But I will need more time to check it.
Anyway, the rest of this answer is still valid and smart requirements pinning suitable for particular project is a good practice imo.
Since I encountered similar issue I agree this is quite annoying. Backtracking might be useful feature but you don't want to wait hours to complete with uncertain success.
I found several option that might help:
--use-deprecated=legacy-resolver
) proposed in the answer by @Daniel Davee, but this is more like temporary solution than a proper one.--no-deps
option. I would not recommend this generally but in some cases you can have a working set of packages versions although there are some conflicts.numpy
in my requirements.txt, I could try numpy >= 1.18.0
or be even more strict with numpy == 1.18.0
. The strictness might help a lot.Check the following sources:
I still do not have a proper answer that would always help but the best practice for requirements.txt seems to "pin" package versions. I found pip-tools that could help you manage this even with constrains.txt (but I am in an experimental phase so I can not tell you more).
It seems author of the question was able to fix the issue (something with custom gitlab server) but I would like to extend this answer since it might be useful for others.
After reading and trying I ended up with pinning all my package versions to a specific one. This really should be the correct way. Although everything can still work without it, there might be cases where if you don't pin your dependencies, your package manager will silently install a new version (when it's released) with possible bugs or incompatibility (this happens to me with dask
last this year).
There are several tools which might help you, I would recommend one of these approaches:
pipreqs
pip install pipreqs
and runnning just pipreqs
in your project root (or eventually with --force
flag if your requirements already exists)requirements.txt
with pinned versions based on imports in your project and versions taken from your environmentrequirements.txt
This is really simple tool (you even do not need to write your requirements.txt). It does not allow you to create something complex (might not be a good choice for bigger projects), last week I found one strange behavior (see this) but generally I'm happy with this tool as it usually works perfectly.
pip-tools
There are several other tools commonly used like pip-tools, Pipenv or Poetry. You can read more in Faster Docker builds with pipenv, poetry, or pip-tools or Python Application Dependency Management in 2018 (older but seems still valid to me). And it still seems to me that the best option (although it depends on your project/use case) is pip-tools
.
You can (this is one option, see more in docs):
requirements.in
(the same format as requirements.txt
, it's up to you whether you pin some package dependency or not)pip install pip-tools
and running pip-compile requirements.in
requirements.txt
file where all versions are pinned, it's clear, what is the origin--generate-hashes
optionpipreqs
) at any time create new environment based on this requirements.txt
pip-tools
offer you --upgrade
option to upgrade the final reqspip-sync
tool to update your environment based on requirements.txtThere are few more stuff you can do with it and I really love the integration with pre-commit
. This allows you to use the same requirements as before (just with .in
suffix) and add pre-commit hook that automatically updates requirements.txt
(so you will never experience having different local environment from the generated requirements.txt
which might easily happen when you run something manually).
So they are changing the resolver, this seems to be a bug. What worked for was using the old resolver, by using the flag
--use-deprecated=legacy-resolver
This will work until pip 21.0 apparently.
source https://github.com/pypa/pip/issues/9215
Had similar issue and wanted to report the solution which worked for me. I had to update the pip version using
pip install --upgrade pip
Actually I was building a docker image so I've added the line
RUN pip install --upgrade pip
Right before installing the requirements. In the time of writing this answer the installed pip version is 22.0.3
. Doing so fixed the problem.
I have pinned all of my dependencies. You will have to pin them later or now so better to do it now. I am on pip 20.3.1
If we don't pin any of our dependency, the new dependency resolver algorithm checks every single available version against each dependency from the latest to the initial version tags.
In my case it never finished. The reason was updating djangorestframework
to a version incompatible with django
. I wanted it to install djangorestframework-3.12.4
, which depends on django >= 2.2
, and also had django==2.0.2
in requirements.txt
.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With