How to develop multiple pip packages in a git project?

Question

I'm developing a django project, using git for code management. The main project has a number of apps as submodules, each of which can be used independently, thus are each in separate git repos. All of these apps are in development, and at least one is forked from another project. These apps are installable through pip, when cloning the repo, there are setup.py and README and so forth in the root, then the actual app in a subfolder.

If I pip install the app(s), then the working code will be in a different location to the folder under git management, so every time I change the code I'd need to pip install (or is there another pip command for this?) to update the code where python is looking for it

I could use pip install -e to prevent the above situation. However, then I would need to have each app cloned into separate folders: I can't just clone the apps into project/apps and have project/apps/foo and project/apps/bar as they would both try to drop their setup.py into project/apps. Instead I would need to clone foo into project/apps/foo but then the actual code is in project/apps/foo/foo. This strikes me as ugly and not very django-ish.

Is there some other, prettier way to do what I'm trying to do?

Answer 1

I think what you really want to use is a requirements file (rather than using git submodules at all) coupled with a virtualenv for your project.

With requirements files, you can clone repos directly from a given branch or commit, for instance:

requirements.txt:

Django==1.4.1
South==0.7.6
git+git://github.com/fabric/fabric#egg=Fabric
git+git://github.com/toastdriven/django-tastypie.git@876c0541e2531d3b6e070ffab906f7c6e359c427#egg=django-tastypie

Then you can run pip install -r requirements.txt.

You'll notice from the tastypie example that you can lock your pip install to a particular commit (fine as long as you stay in your virtualenv), which is essentially the same thing a submodule does anyway, but without cluttering your git repo or file structure with packages when they really should be installed to a separate location that you source anyway.