Project organization with Cython and C++

Question

I want to provide my C++ project with a Python interface. Technically, I have decided to use Cython for wrapping the C++ code. Over time, the entire project is meant to become a Python extension module, but at first, this is highly experimental. Gradually, C++ classes need to be exposed to Python.

My question is how to best organize files and build configurations so that Cython-generated and human-written C++ code do not get mixed and the Python extension module is cleanly built seperate from the other targets.

I imagine a directory structure like this for the source files, and some build directory for Cython.

Project/
    src/
        *.h
        *.cpp
    cython/
        Project.pyx
        setup.py

Answer 1

Basically I have 3 folders :

1. CPROJECT, The C++ library : producing a libcproject.so shared object
2. CYPROJECT, The cythonized Python extension : producing the cyproject.so using Cython
3. DEPENDENCIES, The dependencies : where I copy external requirements for both projects

In 1. I build the C++ extension (compiled with gcc - -shared, -fPIC compile options) that will be exposed to python and that the CYPROJECT relies on to expose features to Python. As a post processing command, the resulting .so is copied into DEPENDENCIES/libcproject/ (as well as the include files). This way the library is, of course, usable independently in a pure C++ project as well.

In 2. I make use of 3 sub-folders :

• adapters : which mainly contains C++ additional classes (often classes derived from the ones provided by libcproject.so). Those are usually classes that are enhanced with functionalities specific to Cython requirements (such as storing the PyObject * C version of a targeted Python version - inherited from object - of a given class and the reference counting management, via Py_XINCREF and Py_DECREF, ...).
• pyext : where are stored all the Cython hand written .pyx files.
• setup : containing the setup.sh script (for setting up the dependencies paths and calling the python setup.py build_ext --inplace for generating the final cyproject.so (to be added to the PYTHONPATH) and cyproject.pyx.

So what's in the setup sub-folder ?

Here is a sample code for setup.sh :

export PYTHONPATH=$PYTHONPATH:../../../DEPENDENCIES/Cython-0.18
export PATH=$PATH:../../../DEPENDENCIES/libcproject:../../../DEPENDENCIES/Cython-0.18/bin

# Note the `../../../DEPENDENCIES/libcproject`...

CC="gcc"   \
CXX="g++"   \
    python setup.py build_ext --inplace

And here an example of setup.py (mainly to demonstrate how the additional adapters are compiled):

import sys
import os
import shutil

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

# Cleaning
for root, dirs, files in os.walk(".", topdown=False):
    for name in files:
        if (name.startswith("cyproject") and not(name.endswith(".pyx"))):
            os.remove(os.path.join(root, name))
    for name in dirs:
        if (name == "build"):
            shutil.rmtree(name)

# Building
setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [
    Extension("cyproject", 
              sources=["cyproject.pyx", \
                       "adapter/ALabSimulatorBase.cpp", \
                       "adapter/ALabSimulatorTime.cpp", \
                       "adapter/ALabNetBinding.cpp", \
                       "adapter/AValueArg.cpp", \
                       "adapter/ALabSiteSetsManager.cpp", \
                       "adapter/ALabSite.cpp", \
                       ],
              libraries=["cproject"],
              language="c++",
              extra_compile_args=["-I../inc", "-I../../../DEPENDENCIES/python2.7/inc", "-I../../../DEPENDENCIES/gsl-1.8/include"], 
              extra_link_args=["-L../lib"]
              extra_compile_args=["-fopenmp", "-O3"],
              extra_link_args=[]
              )
    ]
)                   

And finally, the main .pyx, that links all the hand written .pyxs of the cython part together [cyproject.pyx] :

include "pyext/Utils.pyx" 
include "pyext/TCLAP.pyx" 
include "pyext/LabSimulatorBase.pyx"
include "pyext/LabBinding.pyx"
include "pyext/LabSimulatorTime.pyx"
...

Note : All the files generated by Cython remains in this setup folder, well separated from the hand written stuffs (adapters and pyext), as expected.

In 3. Using a separated DEPENDENCIES folder allows to keep things well separated (in case I would move the CYPROJECT - and its dependencies - in some other environment).

All of this to give you an overview (a pertinent one, I hope) on how one can organize that sort of project.