python2; pip parse_requirements with --trusted-host and --extra-index-url

Question

I am using

install_requires = [str(ir.req) for ir in parse_requirements("requirements.txt", session=PipSession())]

with pip install .. However, this does not seem to work with a requirements.txt that looks like this:

--trusted-host blah
--extra-index-url blah2
...

(support for --trusted-host was added in pip8.0.0). The install from blah fails because it complains about it not being an untrusted host as if it never processed the first line.

HOWEVER, pip install -r requirements.txt works perfectly, so these options are correct.

This means there is something parse_requirements is not doing. My question is: how do I fix or work around this using only pip install .? I could do something horrendous like:

os.system(pip install -r requirements.txt)
setup(...

in the setup.py file.

The implicit coupling of requirements.txt and setup.py is confusing to me. Nothing in setup calls requirements.txt unless you explicitly parse requirements.txt yourself, yet requirements.txt is a very standard python convention.

EDIT: We are using tools (Cloudify and sometimes Chef) that execute a pip install .. We cannot change this. I have to get this working as a pippable package, with the --trusted-host and --extra-index-urls without using a pip.conf either. Currently we are doing the os.system trick.

Answer 1

There has been much written about using setup.py vrs. requirements.txt. Setup.py is for Abstract requirements. Requirements.txt is for concrete requirements. In other words, they serve different purposes. Whereas requirements.txt is for an environment, setup.py is for a package. So it doesn't make sense for a setup.py to read from a requirement.txt just like it wouldn't make sense for a deb package to read from a Chef cookbook. Abstract vrs. Concrete Requirements

Often the reason people do this is they want to support people installing their package with pip install -r requirements.txt from within a check out without needing to list their dependencies twice. That's a reasonable thing to want which is why the requirements file format has a construct that enables it, simply make a requirements.txt file that contains "." or "-e ." and pip will automatically install the project and all of it's dependencies.

EDIT: Since pip is not a library, using the most exposed part of the program is the safest (in my opinion). An alternative to os.system is

import pip
pip.main(['install','-r','requirements.txt'])

Answer 2

This answer from goCards is perfectly valid and you should probably import pip from your setup.py if there's no way around pip install .. But I would like to explain what actually happens here. Here's what you need to know:

1. install_requires is an option only supported by setuptools, a third-party package that improves upon distutils (the standard tool used in setup.py files and distributed with Python).
2. By design, setuptools only accepts actual requirements in install_requires, so options such as --trusted-host cannot be sent to install_requires.
3. This is why you're using parse_requirements("requirements.txt", session=PipSession()). This function only yields packages. The option lines such as --trusted-host bla are not returned, but sent to a PackageFinder if you gave one to parse_requirements. But you don't want these options to be returned, because setuptools would not know what do with those!

Long story short, if you want to use pip options, you need to use pip.