Proper way to parse requirements file after pip upgrade to pip 10.x.x?

Question

So today I did found out that with the release of pip 10.x.x the req package changed its directory and can now be found under pip._internal.req.

Since it is common practice to use the parse_requirements function in your setup.py to install all the dependencies out of a requirements file I now wonder if this practice should change since it is now lying under _internal?

Or what is actually best practice without using parse_requirements?

Answer 1

First, I believe parsing requirements.txt from within setup.py is not a good idea. It should be the other way around, install_requires in setup.py or setup.cfg should be considered as some kind of source of truth, and files such as requirements.txt should be generated from there. But everyone has different needs, that lead to different workflows.

So with that said...

It is possible to parse a relatively simple requirements.txt file from a setuptools setup.py script without pip. The setuptools project already contains necessary tools in its top level package pkg_resources.

It could more or less look like this:

#!/usr/bin/env python

import pathlib

import pkg_resources
import setuptools

with pathlib.Path('requirements.txt').open() as requirements_txt:
    install_requires = [
        str(requirement)
        for requirement
        in pkg_resources.parse_requirements(requirements_txt)
    ]

setuptools.setup(
    install_requires=install_requires,
)

Again, this will work only for simple requirements.txt files. See Requirements parsing in the documentation page for pkg_resources to get details about what is handled. In short, each line should be a valid PEP 508 requirement. Notations that are really specific to pip are not supported and it will cause a failure.

---

Word of caution

As stated already, this is not recommended. The requirements.txt file and the list of "install dependencies" are two different concepts, they are not interchangeable.

But if you insist on writing a setup.py install script that reads a requirements.txt file then make sure that this requirements.txt file is included in the "source distribution" (sdist) otherwise installation will fail, for obvious reasons.

---

Since setuptools version 62.6 it is possible to write something like this in setup.cfg:

[options]
install_requires = file: requirements.txt

Alternatively in pyproject.toml:

[project]
dynamic = ["dependencies"]

[tool.setuptools.dynamic]
dependencies = requirements.txt

Same words of caution as above apply:

• only very simple files are supported
• the file must be added to the sdist

Additionally, it is considered a "beta" feature for now.

---

Notes:

• See also this other answer: https://stackoverflow.com/a/59971469/11138259
• https://caremad.io/posts/2013/07/setup-vs-requirement/
• https://setuptools.pypa.io/en/latest/history.html#v62-6-0

Answer 2

The solution of Scrotch only works until pip 19.0.3, in the pip >= 20 versions the PipSession module was refactored. Here is a solution for the imports that works for all pip versions:

try:
    # pip >=20
    from pip._internal.network.session import PipSession
    from pip._internal.req import parse_requirements
except ImportError:
    try:
        # 10.0.0 <= pip <= 19.3.1
        from pip._internal.download import PipSession
        from pip._internal.req import parse_requirements
    except ImportError:
        # pip <= 9.0.3
        from pip.download import PipSession
        from pip.req import parse_requirements

Answer 3

EDIT: modified to support pip>= 19.0.3

I don't agree with the accepted answer. The setup.py file can get ugly real fast if you have a large project with a lot of dependencies. It is always good practice to keep your requirements in a separate .txt file. I would do something like this -

try:
    # pip >=20
    from pip._internal.network.session import PipSession
    from pip._internal.req import parse_requirements
except ImportError:
    try:
        # 10.0.0 <= pip <= 19.3.1
        from pip._internal.download import PipSession
        from pip._internal.req import parse_requirements
    except ImportError:
        # pip <= 9.0.3
        from pip.download import PipSession
        from pip.req import parse_requirements

requirements = parse_requirements(os.path.join(os.path.dirname(__file__), 'requirements.txt'), session=PipSession())

if __name__ == '__main__':
    setup(
        ...
        install_requires=[str(requirement.requirement) for requirement in requirements],
        ...
    )

Throw in all your requirements in requirements.txt under project root directory.