pip not properly resolving child/grandchild dependencies

Question

I have a dependency tree of modules that works like this (→ indicating a dependency):

a → b, c
b → ruamel.yaml >= 0.16.5
c → ruamel.yaml < 0.16.6, >=0.12.4

It's very clear to me that ruamel.yaml 0.16.5 will resolve both of these dependencies correctly. However, when I pip install a, I get the following logs:

Collecting ruamel.yaml>=0.16.5
  Downloading ruamel.yaml-0.16.10-py2.py3-none-any.whl (111 kB)

And then later:

ERROR: <package c> 0.4.0 has requirement ruamel.yaml<0.16.6,>=0.12.4, but you'll have ruamel-yaml 0.16.10 which is incompatible.

So pip has completely ignored the grandchild dependencies when choosing which packages to install. But it realises that it has messed up at the end. Why is pip not choosing the correct package here. Is there a way to help it work better?

Answer 1

I believe this is a well known problem that is currently being worked on. Message from one week ago: http://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html

In the meantime there are some measures that can be taken to try and mitigate this kind of issues:

• Revert the order of the dependencies (in your example a could list c before b)
• Use an additional requirements.txt or constraints.txt file
• Depending on the actual needs, an alternative tool could help (I believe poetry, pipenv, and most likely others as well might have better dependency resolvers, but they are not a one-to-one replacement for pip)

It appears to be already possible to test pip's future dependency resolver today:

• Install pip from source
• Run path/to/python -m pip install --unstable-feature=resolver ...

In a way it also seems to be possible to somewhat test this dependency resolver in current releases or pip via the pip check command.

Some more references on the topic:

• https://pradyunsg.me/blog/2020/03/27/pip-resolver-testing/
• https://discuss.python.org/t/an-update-on-pip-and-dependency-resolution/1898/2