setup.py & pip: override one of the dependency's sub-dependency from requirements.txt

Question

I'm currently working on a package and in my requirements.txt, I have a dependency: wikipedia. Now, wikipedia 1.3 uses requests-2.2.1 while my package uses version 2.3.0.

Also, as one would expect, wikipedia-1.3's installation depends on presence of it's dependency.

But, If I start a new virtualenv and directly include wikipedia in my requirements.txt, it gives an ImportError on requests since at the time setup.py runs, requests-2.3.0's setup.py doesn't execute unless all others execute. In the Figure attached below, there's no running setup.py for requests after it gets unpacked.

For some weird reason, wikipedia's setup.py contains import wikipedia, which in turn imports it's dependencies before they're even installed; however it passes the CI test because it's installing requirements separately through pip and then running setup.py.

To over come this situation, I've made a setup script consisting of:

pip install -r requirements.txt
pip install wikipedia
pip install -e .

• This installs requests-2.3.0 and beautifulsoup4;
• then installs wikipedia (which can then run setup.py and installs wikipedia and requests-2.2.1)
• then 'pip install -e .' option installs my package along with requests-2.3.0 again.

Hence requests-2.3.0 is first getting installed, then getting replaced by older version 2.2.1 and then replaced again by 2.3.0.

I tried going through various specifications on how to overcome this but those were confusing. How could I overcome this mess?

Answer 1

As noted by Martijn the correct way would be to specify a minimum version in the project assuming full compatibility is preserved in future releases of the sub-dependency.

If you do not have any way to change the requirements file you can download the project and edit the requirements file locally to specify whatever version you want. This can be done via the pip download command:

pip download wikipedia==1.3

Besides that if you want to use pip for the whole process and preserve requests==2.3.0 without deleting and reinstalling again you can specify a constraints file. This can be done with:

pip install -c constraints.txt wikipedia==1.3

Where constraints.txt contains something like:

requests>=2.3.0
beautifulsoup4

This will produce a warning, but the wikipedia package will be installed:

wikipedia 1.3.0 has requirement requests==2.2.1, but you'll have requests 2.3.0 which is incompatible.
Installing collected packages: wikipedia
Successfully installed wikipedia-1.3.0

Now, if you really know what you are doing(or just want to try if it works) you can use the --no-deps flag which will ignore package dependencies entirely and will not produce the warning above:

pip install --no-deps -c constraints.txt wikipedia==1.3

In both cases pip freeze shows:

beautifulsoup4==4.6.0
bs4==0.0.1
requests==2.3.0
wikipedia==1.3.0

Note: This was tested with pip 10.0.1, but it should work with any recent pip version.