How to use autotools to build Python interface at same time as library

Question

I currently have a library written in C++, building with the GNU autotools, and I'd like to add a Python interface to it. Using SWIG I have developed the interface, but I'm having some trouble figuring out how integrate compilation of the Python module in with the rest of the process.

I have looked into AM_PATH_PYTHON but this macro doesn't seem to set the include path for Python.h, so when I compile my module I get a bunch of errors about missing include files. Is there a way to get the Python include path and ldflags out of AM_PATH_PYTHON?

Just for the record I don't think it will be possible to use Python's distutils method (setup.py) as this requires the location of the library in order to link the new module. Since the library has not yet been installed at compile time, I would have to use a relative path (e.g. ../src/lib.so) which of course would break once the Python module was installed (as the library is then in /usr/lib or /usr/local/lib instead.)

EDIT:

Now it can find the .h file it's compiling, but after installing it (in the correct location) Python can't load the module. The code produces foo.so, and when I "import foo" I get this:

ImportError: dynamic module does not define init function (initfoo)

If however I rename it from foo.so to _foo.so then it loads and runs fine, except I have to "import _foo" which I'd rather not have to do. When I follow the SWIG instructions to produce _foo.so in the current directory "import foo" works, so I am not sure why it breaks when the library is installed in the site directory.

EDIT2:

Turns out the problem was I forgot to copy foo.py produced by SWIG into the install directory alongside _foo.so. Once I did this everything worked as expected! Now I just have to figure out some automake rules to copy a file into the install dir...

Answer 1

To find the include path, I'd use python-config. The trick is to use the python-config corresponding to the python installed in $PYTHON.

AM_PATH_PYTHON
AC_ARG_VAR([PYTHON_INCLUDE], [Include flags for python, bypassing python-config])
AC_ARG_VAR([PYTHON_CONFIG], [Path to python-config])
AS_IF([test -z "$PYTHON_INCLUDE"], [
  AS_IF([test -z "$PYTHON_CONFIG"], [
    AC_PATH_PROGS([PYTHON_CONFIG],
                  [python$PYTHON_VERSION-config python-config],
                  [no],
                  [`dirname $PYTHON`])
    AS_IF([test "$PYTHON_CONFIG" = no], [AC_MSG_ERROR([cannot find python-config for $PYTHON.])])
  ])
  AC_MSG_CHECKING([python include flags])
  PYTHON_INCLUDE=`$PYTHON_CONFIG --includes`
  AC_MSG_RESULT([$PYTHON_INCLUDE])
])

Another alternative is to poke around in the distutils.sysconfig module (this has nothing to do with using distutils to build your code). Run python -c "import distutils.sysconfig; help(distutils.sysconfig)" and have a look.

Answer 2

Here is the autoconf macro I call from my ``configure.ac` to find the Python include directory (PYTHONINC) and the Python installation directory (via AM_PATH_PYTHON).

AC_DEFUN([adl_CHECK_PYTHON], 
 [AM_PATH_PYTHON([2.0])
  AC_CACHE_CHECK([for $am_display_PYTHON includes directory],
    [adl_cv_python_inc],
    [adl_cv_python_inc=`$PYTHON -c "from distutils import sysconfig; print sysconfig.get_python_inc()" 2>/dev/null`])
  AC_SUBST([PYTHONINC], [$adl_cv_python_inc])])

Then my wrap/python/Makefile.am builds two Swig modules using Libtool like this :

SUBDIRS = . cgi-bin ajax tests

AM_CPPFLAGS = -I$(PYTHONINC) -I$(top_srcdir)/src $(BUDDY_CPPFLAGS) \
              -DSWIG_TYPE_TABLE=spot

EXTRA_DIST = spot.i buddy.i
python_PYTHON = $(srcdir)/spot.py $(srcdir)/buddy.py
pyexec_LTLIBRARIES = _spot.la _buddy.la

MAINTAINERCLEANFILES = \
  $(srcdir)/spot_wrap.cxx $(srcdir)/spot.py \
  $(srcdir)/buddy_wrap.cxx $(srcdir)/buddy.py

## spot

_spot_la_SOURCES = $(srcdir)/spot_wrap.cxx $(srcdir)/spot_wrap.h
_spot_la_LDFLAGS = -avoid-version -module
_spot_la_LIBADD = $(top_builddir)/src/libspot.la

$(srcdir)/spot_wrap.cxx: $(srcdir)/spot.i
        $(SWIG) -c++ -python -I$(srcdir) -I$(top_srcdir)/src $(srcdir)/spot.i


$(srcdir)/spot.py: $(srcdir)/spot.i
        $(MAKE) $(AM_MAKEFLAGS) spot_wrap.cxx

## buddy

_buddy_la_SOURCES = $(srcdir)/buddy_wrap.cxx
_buddy_la_LDFLAGS = -avoid-version -module $(BUDDY_LDFLAGS)

$(srcdir)/buddy_wrap.cxx: $(srcdir)/buddy.i
        $(SWIG) -c++ -python $(BUDDY_CPPFLAGS) $(srcdir)/buddy.i

$(srcdir)/buddy.py: $(srcdir)/buddy.i
        $(MAKE) $(AM_MAKEFLAGS) buddy_wrap.cxx

The above rules are such that the result of Swig is considered as a source file (i.e. distributed in the tarball) in the same way as a Bison-generated parser would be. This way the final user do not need Swig installed.

When you run make, the *.so files are hidden by Libtool in some .libs/ directory, but after make install they get copied to the right place.

The only trick is how to use the modules from inside the source directory before running make install. E.g. while running make check. For that case, I generate (with configure) a script called run that sets PYTHONPATH prior to running any python script, and I execute all my tests cases through this run script. Here is the contents of run.in, before configure substitutes any value:

# Darwin needs some help in figuring out where non-installed libtool
# libraries are (on this platform libtool encodes the expected final
# path of dependent libraries in each library).
modpath='../.libs:@top_builddir@/src/.libs:@top_builddir@/buddy/src/.libs'

# .. is for the *.py files, and ../.libs for the *.so.  We used to
# rely on a module called ltihooks.py to teach the import function how
# to load a Libtool library, but it started to cause issues with
# Python 2.6.
pypath='..:../.libs:@srcdir@/..:@srcdir@/../.libs:$PYTHONPATH'

test -z "$1" &&
  PYTHONPATH=$pypath DYLD_LIBRARY_PATH=$modpath exec @PYTHON@

case $1 in
  *.py)
    PYTHONPATH=$pypath DYLD_LIBRARY_PATH=$modpath exec @PYTHON@ "$@";;
  *.test)
    exec sh -x "$@";;
  *)
    echo "Unknown extension" >&2
    exit 2;;
esac

If you want to see all this in action in a real project, you can get it from https://spot.lrde.epita.fr/install.html