Preventing namespace collisions between private and pypi-based Python packages

Question

We have 100+ private packages and so far we've been using s3pypi to set up a private pypi in an s3 bucket. Our private packages have dependencies on each other (and on public packages), and it is (of course) important that our GitLab pipelines find the latest functional version of packages it relies on. I.e. we're not interested in the latest checked in code. We create new wheels only after tests and qa has run against a push to master (which is a long-winded way of explaining that -e <vcs> requirements will not work).

Our setup works really well until someone creates a new public package on the official pypi that shadows one of our package names. We can force our private package to be chosen by increasing the version number so it is higher than the new package on pypi.org - or by renaming our package to something that haven't yet been taken on pypi.org.

This is obviously a hacky and fragile solution, but apparently the functionality is this way by-design.

After the initial bucket setup s3pypi has required no maintenance or administration. The above ticket suggests using devpi but that seems like a very heavy solution that requires administration/monitoring/etc.

GitLab's pypi solution seems to be at individual package level (meaning we'd have to list up to 100+ urls - one for each package). This doesn't seem practical, but maybe I'm not understanding something (I can see the package registry menu under our group as well, but the docs point to the "package-pypi" docs).

We can't be the first small company that has faced this issue..? Is there a better way than to register dummy versions of all our packages on pypi.org (with version=0.0.1, so the s3pypi version will be preferred)?

Answer 1

It might not be the solution for you, but I tell what we do.

1. Prefix the package names, and using namespaces (eg. company.product.tool).
2. When we install our packages (including their in-house dependencies), we use a requirements.txt file including our PyPI URL. We run everything in container(s) and we install all public dependencies in them when we are building the images.