Pip install with requirements.txt from private repo prompting for password

Question

I am trying to install a python package my_package from a private Github repo using pip. I am using Github oauth tokens (aka personal access tokens) for security. My token is stored in the environment variable $API_TOKEN.

From the console if I do: pip install git+https://${API_TOKEN}@github.com/johnf1004/my_package.git then it works perfectly.

According to the documentation, you should be able to use environment variables inside requirements.txt. However if I do pip install -r requirements.txt where the file has the exact same address as above (git+https://${API_TOKEN}@github.com/johnf1004/my_package.git) then it fails with a password prompt. FWIW, even a correct password for my Github account still results in failure.

Am I formatting the address/env variable wrong in the requirements file, or what am I missing?

Answer 1

Rather than encoding your credentials (either directly or via an environment variable) into your requirements.txt file, you should configure a credentials helper for git.

Leave the bare URL in your requirements.txt file:

git+https://github.com/johnf1004/my_package.git

Configure an appropriate credentials helper, such as the gh cli. In recent versions, you can run gh auth setup-git to create the necessary configuration.

Run gh auth login to authenticate to GitHub:

$ gh auth login
? What account do you want to log into? GitHub.com
? What is your preferred protocol for Git operations? SSH
? Upload your SSH public key to your GitHub account? Skip
? How would you like to authenticate GitHub CLI? Login with a web browser

! First copy your one-time code: ABCD-1234
Press Enter to open github.com in your browser...
Opening in existing browser session.
✓ Authentication complete.
- gh config set -h github.com git_protocol ssh
✓ Configured git protocol
✓ Logged in as larsks

Now git will get credentials for https://github.com URLs from gh without requiring any interaction on your part.