Pip priority order with index-url and extra-index-url

Question

I've been looking into configuring pip with two indexes: a private PyPI repo (with higher priority) and the public PyPI. The goal is to avoid security risks like dependency confusion/code injection.

Suppose I configure:

index-url = http://my_private_pypi_repo
extra-index-url = https://pypi.org/simple

Questions:

• If a package (e.g., lib) exists in both indexes, which one will pip install lib use?
• If I run pip install lib==0.0.2, but only version 0.0.1 exists in the private index, will pip look at PyPI for 0.0.2?
• What’s the best way to ensure some packages are only fetched from the private index, even if a newer version exists on PyPI?

Answer 1

The short answer is: there is no prioritization and you probably should avoid using --extra-index-url entirely.

---

This is asked and answered here: https://github.com/pypa/pip/issues/5045#issuecomment-369521345

Question:

I have this in my pip.conf:

[global]
index-url = https://myregistry-xyz.com
extra-index-url = https://pypi.python.org/pypi

Let's assume packageX exists in both registries and I run pip install packageX.

I expect pip to install packageX from https://myregistry-xyz.com, but pip will use https://pypi.python.org/pypi instead.

If I switch the values for index-url and extra-index-url I get the same result. pypi is always prioritized.

Answer:

Packages are expected to be unique up to name and version, so two wheels with the same package name and version are treated as indistinguishable by pip. This is a deliberate feature of the package metadata, and not likely to change.

---

I would also recommend reading this discussion: https://discuss.python.org/t/dependency-notation-including-the-index-url/5659

There are quite a lot of things that are addressed in this discussion, some that is clearly out of scope for this question, but everything is very informative anyway.

In there, there should be the key takeaway for you:

Pip does not really prioritize one index over the other in theory. In practice, because of a coincidence in the way things are implemented in code, it might be that one is always checked first, but it is not a behavior you should rely on.

And what is a good way to be in control, that certain libraries will only be fetched from the private index if they exists there, and will not be looked for at PyPI?

You should setup and curate your own package index (devpi, pydist, jfrog artifactory, sonatype nexus, etc.) and use it exclusively, meaning: never use --extra-index-url. This is the only way you can have exact control over what gets downloaded. This custom repository might function mostly a proxy for the public PyPI, except for a couple of dependencies.

---

For a potential solution to some of the reasons that lead to ask about index priority order, keep an eye on "PEP 708 – Extending the Repository API to Mitigate Dependency Confusion Attacks"

---

Related:

• pip: selecting index url based on package name?
• https://discuss.python.org/t/24179