Using custom packages on my python project

Question

I'm doing a few projects in python right now, and I'm trying to figure out how to work with my own versions of existing open source packages.

For instance, I'm using tipfy with zc.buildout, and I've added in the 'paypal' package. Unfortunately it doesn't have a feature I need, so I've forked it on github and added the feature. I will send the original package maintainers a pull request, but whether they accept my additions or not, I'd like to use my version of the package and keep the convenience of having zc.buildout manage my dependencies. How do I do this?

Do I upload my own take on the library to PyPI and prefix it with my name? Wouldn't that unnecessarily pollute the index?

Or should I make and maintain my own index and package repo? Where do I find the format for this? And is it against to terms of the OSS licenses to host my own repo with modified packages with the same names? (I'd rather not modify every file in the project with new namespaces)

I'm sure this problem comes up quite a lot, and not just with python. I can see this happening with Maven and SBT, too... what do people usually do when they want to use their own versions of popular packages?

Answer 1

There are two ways you could go about this. I use both, depending on the context the buildout is being used for:

1. Use mr.developer to include packages from a version control system (mr.developer support a wide range of systems, including git). I use this when developing.

2. Build a private package repository. A simple directory listing in Apache is enough. Add the URL to your private repository as a find-links entry:

   [buildout]
   ...
   find-links =
       ...
       http://username:[email protected]/projectname
   

   In this example I also included a username and password (buildout then will authenticate) and a path on the server that is project specific. You can also just build one big private repository for all your projects, of course.

   In that repository you either put complete eggs, or just a tarball of the package. Repositories named in find-links are searched before PyPI.

   I use this method for deployment buildouts. This way the software will use released packages, which makes release management much clearer and simpler.

Hosting your own forks of OSS packages is perfectly fine! This is one of the freedoms you get when using OSS. Note that when you fork GPL code like this, and are distributing it to a third party, you need to make your changes available to them. A package repository is one way of doing complying with this.

Answer 2

To keep your headache in check, I would really recommend just bundling all such custom code with your package. Say you made a fork of some package. As long as its license allows it, I would just bundle the modified version of package with my code, as if it's just another directory. You can place it locally under package so it will be easily found. Once the developers of package fix what you need, just remove this directory and make it a dependency on an online package once again.

An added bonus of this approach is making distribution to users/customers easier.

Answer 3

It's been a while since I've used buildout, but if I recall correctly, there is a pip recipe that allows you to use pip requirements files. You can create a requirements file that contains something like this:

-e git+http://<github url>

That will check the package out locally when installing.