pip3 setup.py install_requires PEP 508 git URL for private repo

Question

I am trying to run:

pip3 install -e .

in my Python project where I have the following setup.py:

from setuptools import setup

setup(
    name='mypackage',
    install_requires=[
        "anotherpackage@[email protected]:myorg/anotherpackage.git"
    ]
)

but it fails with:

error in mypackage setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid URL given

I guess it is correct about the format of my URL as PEP 508 doesn't allow specifying git user name for ssh clone URLs.

What is the correct syntax for PEP 508 URLs with git+ssh protocol for install_requires dependency for private git repositories (in this case hosted on BitBucket)? What is the syntax for specifying a specific branch, tag or sha?

More context to avoid XY problem

I have an internal Python project that depends on multiple internally developed Python packages. I would like to avoid the necessity for hosting my own PIP repository in the organisation and thus I am trying to use git URLs directly. I need to use ssh protocol for git URLs as all the users have their ssh keys configured and it would be cumbersome to ask all the users to configure their app passwords in BitBuckets (I have 2FA required and the regular user password doesn't work).

I have already tried to use:

dependency_links

setup(
    name='mypackage',
    install_requires=[
        "anotherpackage==0.0.1"
    ],
    dependency_links=[
        "[email protected]:myorg/[email protected]#egg=anotherpackage-0.0.1"
    ]
)

But they are deprecated and they are ignored by pip3 install -e .. According to documentation I've found, PEP 508 URLs should be used instead.

requirements.txt file with entries duplicated from install_requires entries

I have a requirements.txt file with:

-e [email protected]:myorg/[email protected]#egg=anotherpackage

and I use pip3 install -r requirements.txt instead of pip3 install -e .. It works but is suboptimal as I have to keep both setyp.py and requirements.txt in sync.

If there is any other recommended solution for my problem I would like to learn about it :)

Answer 1

After checking pip source code I found the correct syntax for private BitBucket repositories.

The general form for the packages with URLs is <package name>@<URI> and the URI must start with a <scheme>://.

So I fixed it to:

anotherpackage@git+ssh://[email protected]:myorg/anotherpackage.git

and then I was getting a different error - this time git command (invoked by pip) was complaining about repository URL ssh://[email protected]:myorg/anotherpackage.git.

I checked the git documentation for the ssh:// URLs format and found out that hostname and organisation parts must be separated with / instead of ::

ssh://[email protected]/myorg/anotherpackage.git

This URL works fine. I also learned from the pip source code that the actual revision/branch/tag can be specified by appending @<rev-spec> so I can specify for example the tag 0.0.1 with the following in install_requires:

anotherpackage@git+ssh://[email protected]:myorg/[email protected]

The only issue that I still have is that when I change the revision and run pip3 install -e . again it doesn't detect the change (even when run with --upgrade). I have to manually uninstall the package (pip3 uninstall anotherpackage) and run pip3 install -e . again.