dealing with pip install dependency conflicts

Question

I have ~300 python packages that I am trying to install from a shell script that is configured to run when an instance is created. The script fails due to dependency conflicts.

ERROR: Cannot install ipykernel==6.16.2, jupyter-client==7.3.4, nbclassic==0.4.5, nbclient==0.7.0, nest-asyncio==1.5.6, notebook==6.5.1 and sparkmagic==0.20.0 because these package versions have conflicting dependencies.

2023-02-09T21:47:54.127-08:00

Copy
The conflict is caused by:
    The user requested nest-asyncio==1.5.6
    ipykernel 6.16.2 depends on nest-asyncio
    jupyter-client 7.3.4 depends on nest-asyncio>=1.5.4
    nbclassic 0.4.5 depends on nest-asyncio>=1.5
    nbclient 0.7.0 depends on nest-asyncio
    notebook 6.5.1 depends on nest-asyncio>=1.5
    sparkmagic 0.20.0 depends on nest-asyncio==1.5.5
The conflict is caused by: The user requested nest-asyncio==1.5.6 ipykernel 6.16.2 depends on nest-asyncio jupyter-client 7.3.4 depends on nest-asyncio>=1.5.4 nbclassic 0.4.5 depends on nest-asyncio>=1.5 nbclient 0.7.0 depends on nest-asyncio notebook 6.5.1 depends on nest-asyncio>=1.5 sparkmagic 0.20.0 depends on nest-asyncio==1.5.5

2023-02-09T21:47:54.127-08:00

Copy
To fix this you could try to:


2023-02-09T21:47:54.127-08:00

Copy
1. loosen the range of package versions you've specified

2023-02-09T21:47:54.127-08:00

Copy
2. remove package versions to allow pip attempt to solve the dependency conflict

2023-02-09T21:47:58.887-08:00

Copy
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

How do deal with conflicts without having to individually update the version numbers for each of them? I obtain all the packages by running pip freeze command.

Answer 1

Package dependency resolvers like poetry or pip-tools are specifically designed to solve this problem.

Poetry is featureful, user-friendly, and its documentation is quite good: https://python-poetry.org/docs/basic-usage/

pip-tools is lightweight, straightforward, and "according to PEP specs": https://github.com/jazzband/pip-tools