distutils, distutils2, pip and requirements [closed]

Question

I am diving into the world of packaging Python applications and managed to get into this state of confusion where my head starts to spin due to all the concepts and options I am supposed to deal with.

Question:

What do I need to accomplish? Deploy my Python project from source located on a git server. The deployment tool should get and install all the dependencies, most of which are available via PIP and one needs to be fetched and installed via Git. The final result should be installable via Pip, so I can do something as:

[~] git clone git://some/path/project.git
[~] pip install project/

Context: Currently I am trying to get Distutils2 to do what I want, but it seems that setup.py made using 'generate-setup' command doesn't play along with Pip.

I wanted to use Distutils2 since it is supposed to be most future proof of all. But various documentation on all the tools is just horrible (accurate info mixed with outdated and inaccurate information) in a way that makes a guy question his sanity.

So what should I do? Stick to distutils and setup.py? Or do I need to take a look at something the likes of Buildout?

Could the kind answerer please lay out what I am supposed to do with particular tool (e.g.: deploy your code using Distutils2, install dependencies using PIP, for git dependencies write a script and glue everything together doing XYZ).

Edit: I am using Distutils2 1.0a4, which seems incompatible with the docs.

Edit2: Reformatted the question to make it clearer what my question is really about.

Edit3: This is my fourth attempt at breaking the packaging and distribution toolchain of Python. I am not trying to get other people to do my work for me, however for a rookie it is pretty much impossible to crack what a particular tool is supposed to do, where it starts and where another ends. Especially because of the functional overlap between the tools. I am not located in Silicon Valley encircled by sages who could initiate me into the secrets and the publicly available documentation is of no use.

Final Edit: Although I wasn't really thinking about replacing virtualenv with Buildout when starting this question. But while doing my research I realized something I always knew, but just didn't come to me in total clarity. There are many ways about Python packaging and deployment automation. There are also many tools which can help you get the stuff done. However while there is significant functional overlap between the tools, the toolchain is ever evolving and thus far there is no clear "standard best practice". The distribution toolchain arms race is still in full heat and no clear victor has emerged yet. This may be confusing to us noobs, who expect that most of shit in Python just works. What I was after (distutils/setuptools + pip + virtualenv in a Buildout fashion or even semi integrated with Buildout) certainly is dooable, but it just doesn't make much sense, not because its not possible - but because nobody does it. If you need this level of sophistication, then you need to commit. Personally I have decided to leave virtualenv behind (for this project) and embrace Buildout.

Answer 1

Take a look at buildout; together with a buildout plugin called mr.developer you can put together a deployment system that'll do all that you ask for.

There are plenty of examples and presentations on buildout configurations on the web, here are a few to get you started:

• An introductionairy presentation on buildout: http://www.slideshare.net/djay/buildout-how-to-maintain-big-app-stacks-without-losing-your-mind

  Includes a YouTube video of the presentation, so you can listen along.

• Excellent blog post on how to use buildout to develop a Django application.

  Includes details on how buildout and setup.py interplay.

• The configuration for planet.plone.org: https://github.com/plone/planet.plone.org/blob/master/buildout.cfg

  This builds a venus RSS aggregator with configuration, style, apache config and cronjobs, pulling in eggs as needed.

• The Plone core development buildout: https://github.com/plone/buildout.coredev

  Complex buildout that pulls in all the sources needed to develop the Plone CMS; this is a complex beast but it shows off what you can do with mr.developer.