How to organize Python modules for PyPI to support 2.x and 3.x

Question

I have a Python module that I would like to upload to PyPI. So far, it is working for Python 2.x. It shouldn't be too hard to write a version for 3.x now.

But, after following guidelines for making modules in these places:

• Distributing Python Modules
• The Hitchhiker’s Guide to Packaging

it's not clear to me how to support multiple source distributions for different versions of Python, and it's not clear if/how PyPI could support it. I envisage I would have separate code for:

• 2.x
• 2.6 (maybe, as a special case to use the new buffer API)
• 3.x

How is it possible to set up a Python module in PyPI so that someone can do:

easy_install modulename

and it will install the right thing whether the user is using 2.x or 3.x?

Answer 1

I found that setup.py for httplib2 seems to have an elegant way to support Python 2.x and 3.x. So I decided to copy that method.

The task is to craft a single setup.py for the package distribution that works with all the supported Python distributions. Then with the same setup.py, you can do:

python2 setup.py install

as well as

python3 setup.py install

It should be possible to keep setup.py simple enough to be parsed with all the supported Python distributions. I've successfully done so with a package cobs that supports 2.4 through 2.6 as well as 3.1. That package includes pure Python code (separate code for Python 2.x and 3.x) and C extensions, written separately for 2.x and 3.x.

To do it:

1) I put the Python 2.x code into a python2 subdirectory, and Python 3.x code in a python3 subdirectory.

2) I put the C extension code for 2.x and 3.x in a src directory under python2 and python3.

So, the directory structure is:

root
  |
  +--python2
  |     |
  |     +--src
  |
  +--python3
  |     |
  |     +--src
  |
  +--setup.py
  +--MANIFEST.in

3) In the setup.py, I had these lines near the top:

if sys.version_info[0] == 2:
    base_dir = 'python2'
elif sys.version_info[0] == 3:
    base_dir = 'python3'

4) In the call to setup, I specified the packages as normal:

setup(
    ...
    packages=[ 'cobs', 'cobs.cobs', 'cobs.cobsr', ],

5) I specified the base directory for the Python code using a package_dir option (refer to step 3 for base_dir):

    package_dir={
        'cobs' : base_dir + '/cobs',
    },

6) For the C extensions, I gave the path:

    ext_modules=[
        Extension('cobs.cobs._cobs_ext', [ base_dir + '/src/_cobs_ext.c', ]),
        Extension('cobs.cobsr._cobsr_ext', [ base_dir + '/src/_cobsr_ext.c', ]),
    ],

That was about it for setup.py. The setup.py file is parsable by both Python 2.x and 3.x.

7) Finally, if you build a source distribution using:

python2 setup.py sdist

then it will by default pull in only the files that are specifically needed to build for that Python. E.g. in the above case, you would only get the files under python2 in the source distribution, but not those under python3. But for a complete source distribution, you want to include the files for both 2.x and 3.x. To do that, create a MANIFEST.in file that contains something like this:

include *.txt
recursive-include python2 *
recursive-include python3 *

To see what I did, see the cobs source code on PyPI or BitBucket.

Answer 2

The simplest solution is to use a single source distribution.