Speed up Python Poetry dependancy resolution

Question

The dependency resolution of my Poetry environments frequently takes extremely long (20 minutes or more).

Running poetry lock -vvv I notice that for some packages, sdists for multiple versions will get downloaded, taking several seconds each time.

Additionally, I repeatedly see messages like the following:

This seems to be the bottleneck for resolution.

I also see a message earlier in the logs:

Private PyPi: Response url ... differs from request url ...

I'm not sure if that's related, but I do use a private PyPi server as my secondary:

[[tool.poetry.source]]
name = "private_pypi"
url = "https://pypi.private_pypi.com.au/simple"
secondary = true

[[tool.poetry.source]]
name = "pypi-public"
url = "https://pypi.org/simple/"

There is an extensive discussion on Poetry Git issue 2094 that suggests many of the resolution woes are out of Poetry's hands. Not sure if this is the case for me.

How might I try to speed things up? For example, will nailing down versions in my pyproject.toml (ie: using == not ^ or >=) help?

Answer 1

TL;DR: Use >= where appropriate. See warnings.

To speed things up, try setting minimum versions in your pyproject.toml as you mentioned. Personally, i like somewhat open-ended dependencies so poetry lock will fix CVEs (security vulnerabilities) with little to no work. However, to greatly speed up dependency resolution, I set the current version (from my successfully resolved poetry.lock / poetry show --tree) as the minimum. Obvious warning that this could break dependencies if others import your code! You're saying "i need this latest version" when in reality you don't and just want faster resolution. So don't do this for open-source code. And either way, add a comment as to why, and what the minimum version really is. Especially if a certain minimum is required due to a CVE, which you should mention in the comment.

For your example, I don't see the value of sdist but from the images you posted, this would mean:

numpy = ">=1.21.2"  # Our code requires minimum version >=[enter here] due to CVE 1234
pandas = ">=1.4.2"  # Minimum version to speed up resolution. Will work with any 1.0.0+

Some people like to use ^ (don't allow for version upgrades, see https://python-poetry.org/docs/dependency-specification/) instead of the open-ended >= min version (no max) but if you have decent unit tests, you should be able to detect any breaking updates.

I would also recommend ONLY listing libraries you import in your own code in pyproject.toml so that you don't end up installing libraries you don't need when a pypi library switches (say pandas stops using numpy, and switched to some shiny new library in this theoretical example).

For example, if you don't import anything from numpy, and it's only installed because of pandas, then only set:

pandas = ">=1.4.2"  # Minimum version to speed up resolution. Will work with any 1.0.0+. This will also set a minimum version for numpy. See `poetry show tree` for more details

Why? Because pandas comes with its own dependencies, and you setting one may later break the tree.

With that warning against setting external library's requirements, there are cases where you may want to force a min dependency just to speed up the resolution process. Imagine the library said numpy = ">= 1.0". Then it's possible that attempting to resolve pandas (in this example) would download the requirements for every version since 1.0.0 before making a decision (especially on failures to resolve).

I was able to get my poetry lock to go from 3 minutes -> 19 seconds by setting two libraries!

Secondary warning: using this method will slowly drift over time and get back to terrible dependency resolution. When that happens, you'll probably want to increase the minimum version (around once a year), depending on how quickly your main external libraries get updated.

Which libraries should you ">= pin"? Those that take the longest (or go through the highest number of versions) in poetry lock -vvv. Play around and see which libraries are worth setting values for vs the amount of time it saves. Avoid setting versions for things that resolved quickly; there are many drawbacks! Note that this is also a problem with pip's requirements.txt and is not unique to poetry.