Best practice for bundling third party libraries for distribution in Python 3

Question

I'm developing an application using Python 3. What is the best practice to use third party libraries for development process and end-user distribution? Note that I'm working within these constraints:

• Developers in the team should have the exact same version of the libraries.
• An ideal solution would work on both Windows and Linux.
• I would like to avoid making the user install software before using our own; that is, they shouldn't have to install product A and product B before using ours.

Answer 1

You could use setuptools to create egg files for your libraries, assuming they aren't available in egg form already. You could then bundle the eggs alongside your software, which would need to either install them, or ensure that they were on the import path.

This has some complexities, i.e. if your libraries have C-extensions, then your eggs become platform-specific, but in my experience this is the most widely-accepted means of 'bundling' stuff in Python.

I have to say that this remains one of Python's weaknesses, though; the third-party ecosystem is certainly aimed at developers rather than end-users.

Answer 2

There are no best practices, but there are a few different tracks people follow. With regard to commercial product distribution there are the following:

Manage Your Own Package Server

With regard to your development process, it is typical to either have your dev boxes update from a local package server. That allows you to "freeze" the dependency list (i.e. just stop getting upstream updates) so that everyone is on the same version. You can update at particular times and have the developers update as well, keeping everyone in lockstep.

For customer installs you usually write an install script. You can collect all the packages and install your libs, as well as the other at the same time. There can be issues with trying to install a new Python, or even any standard library because the customer may already depend on a different version. Usually you can install in a sandbox to separate your packages from the systems packages. This is more of a problem on Linux than Windows.

Toolchain

The other option is to create a toolchain for each supported OS. A toolchain is all the dependencies (up to, but not including base OS libs like glibc). This toolchain gets packaged up and distributed for both developers AND customers. Best practice for a toolchain is:

• change the executable to prevent confusion. (ie. python -> pkg_python)
• don't install in .../bin directories to prevent accidental usage. (ie. on Linux you can install under .../libexec. /opt is also used although personally I detest it.)
• install your libs in the correct location under lib/python/site-packages so you don't have to use PYTHONPATH.
• Distribute the source .py files for the executables so the install script can relocate them appropriately.
• The package format should be an OS native package (RedHat -> RPM, Debian -> DEB, Win -> MSI)