How to distribute and deploy Python 3 code with dependency isolation

Question

I'm not happy with the way that I currently deploy Python code and I was wondering if there is a better way. First I'll explain what I'm doing, then the drawbacks:

• When I develop, I use virtualenv to do dependancy isolation and install all libraries using pip. Python itself comes from my OS (Ubuntu)
• Then I build my code into a ".deb" debian package consisting of my source tree and a pip bundle of my dependancies
• Then when I deploy, I rebuild the virtualenv environment, source foo/bin/activate and then run my program (under Ubuntu's upstart)

Here are the problems:

1. The pip bundle is pretty big and increases the size of the debian package significantly. This is not too big a deal, but it's annoying.
2. I have to build all the C libraries (PyMongo, BCrypt, etc) every time I deploy. This takes a little while (a few minutes) and it's a bit lame to do this CPU bound job on production

Here are my constraints:

1. Must work on Python 3. Preferably 3.2
2. Must have dependency isolation
3. Must work with libraries that use C (like PyMongo)

I've heard things about freezing, but I haven't been able to get this to work. cx_freeze out of Pypi doesn't seem to compile (on my Python, at least). The other freeze utilities don't seem to work with Python 3. How can I do this better?

Answer 1

Wheel is probably the best way to do this at the moment.

Create a virtualenv on the deployment machine, and deploy a wheel along with any dependencies (also built as wheels) to that virtualenv.

This solves the problems:

1. Having separate wheels for dependencies mean you don't have to redeploy dependencies that haven't changed, cutting the size of the deployment artefact
2. Building big packages (such as lxml or scipy) can be done locally, and then the compiled wheel pushed to production

Also, it works fine with libraries that use C.