Why is pip installing an incompatible package version?

Question

I am using pip 20.0.2 on Ubuntu, and installing a bunch of requirements from a requirements file. For some reason, pip is deciding to install idna==2.9(link), even though that is not a compatible version with one of my directly listed dependencies. So I used python -m pipdeptree -r within the virtualenv that I'm installing everything to, and I see this listed for idna:

idna==2.9                                         
  - cryptography==2.3.1 [requires: idna>=2.1]
  - requests==2.22.0 [requires: idna>=2.5,<2.9]
    - requests-oauthlib==1.3.0 [requires: requests>=2.0.0]
      - social-auth-core==3.2.0 [requires: requests-oauthlib>=0.6.1]
        - social-auth-app-django==2.1.0 [requires: social-auth-core>=1.2.0]
    - responses==0.10.9 [requires: requests>=2.0]
    - social-auth-core==3.2.0 [requires: requests>=2.9.1]                           
      - social-auth-app-django==2.1.0 [requires: social-auth-core>=1.2.0]

As we can see, my two direct dependencies (cryptography and requests), are what require idna. According to those, it looks like pip should decide to install 2.8, because it is the latest version that will fulfill the constraints.

Why is pip instead installing idna 2.9, as indicated by the top line of that output, and this error message when running pip install -r requirements.txt:

ERROR: requests 2.22.0 has requirement idna<2.9,>=2.5, but you'll have idna 2.9 which is incompatible.

EDIT: the contents of requirements.txt and it's children, as requested in the comments:

# requirements.txt
-r requirements/requirements-base.txt
-r requirements/requirements-testing.txt

# requirements-base.txt
cryptography~=2.3.1
pyjwt~=1.6.4
requests~=2.22.0
social-auth-app-django~=2.1.0

# requirements-testing.txt
hypothesis~=3.87.0
pytest~=3.6.2
pytest-django~=3.3.2
pytest-cov~=2.5.1
responses~=0.10.5

Edit 2: I've created a minimally viable example. For this example, here is requirements.txt:

cryptography~=2.3.1
requests~=2.22.0

And here are the commands I ran from start to finish in a fresh directory:

virtualenv -p python3.6 -v venv
source venv/bin/activate
pip install -r requirements.txt --no-cache-dir

And the full output:

Collecting cryptography~=2.3.1
  Downloading cryptography-2.3.1-cp34-abi3-manylinux1_x86_64.whl (2.1 MB)
     |████████████████████████████████| 2.1 MB 2.0 MB/s 
Collecting requests~=2.22.0
  Downloading requests-2.22.0-py2.py3-none-any.whl (57 kB)
     |████████████████████████████████| 57 kB 18.5 MB/s 
Collecting asn1crypto>=0.21.0
  Downloading asn1crypto-1.3.0-py2.py3-none-any.whl (103 kB)
     |████████████████████████████████| 103 kB 65.4 MB/s 
Collecting idna>=2.1
  Downloading idna-2.9-py2.py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 71.4 MB/s 
Collecting six>=1.4.1
  Downloading six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting cffi!=1.11.3,>=1.7
  Downloading cffi-1.14.0-cp36-cp36m-manylinux1_x86_64.whl (399 kB)
     |████████████████████████████████| 399 kB 30.3 MB/s 
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
  Downloading urllib3-1.25.8-py2.py3-none-any.whl (125 kB)
     |████████████████████████████████| 125 kB 46.7 MB/s 
Collecting certifi>=2017.4.17
  Downloading certifi-2019.11.28-py2.py3-none-any.whl (156 kB)
     |████████████████████████████████| 156 kB 65.1 MB/s 
Collecting chardet<3.1.0,>=3.0.2
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
     |████████████████████████████████| 133 kB 60.8 MB/s 
Collecting pycparser
  Downloading pycparser-2.19.tar.gz (158 kB)
     |████████████████████████████████| 158 kB 25.0 MB/s 
Building wheels for collected packages: pycparser
  Building wheel for pycparser (setup.py) ... done
  Created wheel for pycparser: filename=pycparser-2.19-py2.py3-none-any.whl size=111031 sha256=030a1449dd5902f2f03e9e2f8f9cc6760503136a9243e965237a1ece1196502a
  Stored in directory: /tmp/pip-ephem-wheel-cache-c_dx8qi5/wheels/c6/6b/83/2608afaa57ecfb0a66ac89191a8d9bad71c62ca55ee499c2d0
Successfully built pycparser
ERROR: requests 2.22.0 has requirement idna<2.9,>=2.5, but you'll have idna 2.9 which is incompatible.
Installing collected packages: asn1crypto, idna, six, pycparser, cffi, cryptography, urllib3, certifi, chardet, requests
Successfully installed asn1crypto-1.3.0 certifi-2019.11.28 cffi-1.14.0 chardet-3.0.4 cryptography-2.3.1 idna-2.9 pycparser-2.19 requests-2.22.0 six-1.14.0 urllib3-1.25.8

Answer 1

Pip does not have a dependency resolver. If you tell it to install package foo without any qualifications, you’re getting the newest version of foo, even if it conflicts with other packages you’ve already installed.

Other solutions like poetry exist which do have the logic to keep everything compatible. If you need this, consider using something like that instead of plain pip.