Cython and fortran - how to compile together without f2py

Question

FINAL UPDATE

This question is about how to write a setup.py that will compile a cython module which accesses FORTRAN code directly, like C would. It was a rather long and arduous journey to the solution, but the full mess is included below for context.

ORIGINAL QUESTION

I have an extension which is a Cython file, which sets up some heap memory and passes it to the fortran code, and a fortran file, which is a venerable old module that I'd like to avoid reimplementing if I can.

The .pyx file compiles fine to C, but the cython compiler chokes on the .f90 file with the following error:

$ python setup.py build_ext --inplace
running build_ext
cythoning delaunay/__init__.pyx to delaunay/__init__.c
building 'delaunay' extension
error: unknown file type '.f90' (from 'delaunay/stripack.f90')

Here's (the top half of) my setup file:

from distutils.core import setup, Extension
from Cython.Distutils import build_ext

ext_modules = [
  Extension("delaunay",
    sources=["delaunay/__init__.pyx",
             "delaunay/stripack.f90"])
]

setup(
  cmdclass = {'build_ext': build_ext},
  ext_modules = ext_modules,
  ...
)

NOTE: I originally had the fortran file's location incorrectly specified (without the directory prefix) but this breaks in exactly the same way after I fixed that.

Things I have tried:

I found this, and tried passing in the name of the fortran compiler (i.e. gfortran) like this:

$ python setup.py config --fcompiler=gfortran build_ext --inplace
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: option --fcompiler not recognized

And I've also tried removing --inplace, in case that was the problem (it wasn't, same as the top error message).

So, how do I compile this fortran? Can I hack it into a .o myself and get away with linking it? Or is this a bug in Cython, which will force me to reimplement distutils or hack around with the preprocessor?

UPDATE

So, having checked out the numpy.distutils packages, I understand the problem a bit more. It seems that you have to

1. Use cython to convert the .pyx files to cpython .c files,
2. Then use an Extension/setup() combination that supports fortran, like numpy's.

Having tried this, my setup.py now looks like this:

from numpy.distutils.core import setup
from Cython.Build import cythonize
from numpy.distutils.extension import Extension

cy_modules = cythonize('delaunay/sphere.pyx')
e = cy_modules[0]

ext_modules = [
  Extension("delaunay.sphere",
      sources=e.sources + ['delaunay/stripack.f90'])
]

setup(
  ext_modules = ext_modules,
  name="delaunay",
  ...
)

(note that I've also restructured the module a bit, since seemingly an __init__.pyx is disallowed...)

Now is where things become buggy and platform-dependent. I have two testing systems available - one Mac OS X 10.6 (Snow Leopard), using Macports Python 2.7, and one Mac OS X 10.7 (Lion) using the system python 2.7.

On Snow Leopard, the following applies:

This means that the module compiles (hurray!) (although there's no --inplace for numpy, it seems, so I had to system-wide install the testing module :/) but I still get a crash on import as follows:

  >>> import delaunay
  Traceback (most recent call last):
    File "<input>", line 1, in <module>
    File "<snip>site-packages/delaunay/__init__.py", line 1, in <module>
      from sphere import delaunay_mesh
  ImportError: dlopen(<snip>site-packages/delaunay/sphere.so, 2): no suitable image found.  Did find:
    <snip>site-packages/delaunay/sphere.so: mach-o, but wrong architecture

and on Lion, I get a compile error, following a rather confusing looking compile line:

gfortran:f77: build/src.macosx-10.7-intel-2.7/delaunay/sphere-f2pywrappers.f
/usr/local/bin/gfortran -Wall -arch i686 -arch x86_64 -Wall -undefined dynamic_lookup -bundle build/temp.macosx-10.7-intel-2.7/delaunay/sphere.o build/temp.macosx-10.7-intel-2.7/build/src.macosx-10.7-intel-2.7/delaunay/spheremodule.o build/temp.macosx-10.7-intel-2.7/build/src.macosx-10.7-intel-2.7/fortranobject.o build/temp.macosx-10.7-intel-2.7/delaunay/stripack.o build/temp.macosx-10.7-intel-2.7/build/src.macosx-10.7-intel-2.7/delaunay/sphere-f2pywrappers.o -lgfortran -o build/lib.macosx-10.7-intel-2.7/delaunay/sphere.so
ld: duplicate symbol _initsphere in build/temp.macosx-10.7-intel-2.7/build/src.macosx-10.7-intel-2.7/delaunay/spheremodule.o ldand :build /temp.macosx-10.7-intelduplicate- 2.7symbol/ delaunay/sphere.o _initsphere in forbuild architecture /i386
temp.macosx-10.7-intel-2.7/build/src.macosx-10.7-intel-2.7/delaunay/spheremodule.o and build/temp.macosx-10.7-intel-2.7/delaunay/sphere.o for architecture x86_64

Now let's just step back a moment before we pore over the details here. Firstly, I know there are a bunch of headaches over architecture clashes in 64-bit Mac OS X; I had to work very hard to get Macports Python working on the Snow Leopard machine (just to upgrade from system python 2.6). I also know that when you see gfortran -arch i686 -arch x86_64 you are sending mixed messages to your compiler. There are all manner of platform-specific problems buried in there, that we don't need to worry about in the context of this question.

But let's just look at this line: gfortran:f77: build/src.macosx-10.7-intel-2.7/delaunay/sphere-f2pywrappers.f

what is numpy doing?! I don't need any f2py features in this build! I actually wrote a cython module in order to avoid dealing with f2py's insanity (I need to have 4 or 5 output variables, as well as neither-in-nor-out arguments - neither of which is well supported in f2py.) I just want it to compile .c -> .o, and .f90 -> .o and link them. I could write this compiler line myself if I knew how to include all the relevant headers.

Please tell me I don't need to write my own makefile for this... or that there's a way to translate fortran to (output-compatible) C so I can just avoid python ever seeing the .f90 extension (which fixes the whole problem.) Note that f2c is not suitable for this as it only works on F77 and this is a more modern dialect (hence the .f90 file extension).

UPDATE 2 The following bash script will happily compile and link the code in place:

PYTHON_H_LOCATION="/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/"

cython sphere.pyx

gcc -arch x86_64 -c sphere.c -I$PYTHON_H_LOCATION
gfortran -arch x86_64 -c stripack.f90
gfortran -arch x86_64 -bundle -undefined dynamic_lookup -L/opt/local/lib *.o -o sphere.so

Any advice on how to make this kind of hack compatible with a setup.py? I don't anyone installing this module to have to go find Python.h manually...

Answer 1

UPDATE: I've created a project on github which wraps up this generating of compile lines by hand. it's called complicated_build.

UPDATE 2: in fact, "generating by hand" is a really bad idea as it's platform specific — the project now reads the values from the distutils.sysconfig module, which is the settings used to compile python (i.e. exactly what we want,) the only setting which is guessed is fortran compiler and file extensions (which are user-configurable). I suspect it is reimplementing a fair bit of distutils now!

---

The way to do this is to write your own compiler lines, and hack them into your setup.py. I show an example below which works for my (very simple) case, which has the following strucutre:

• imports
• cythonize() any .pyx files, so you only have fortran and C files.
• define a build() function which compiles your code:
  • maybe some easy-to-change constants, like compiler names and architecture
  • list up the fortran and C files
  • generate the shell commands that will build the modules
  • add the linker line
  • run the shell commands.
• if the command was install and the target doesn't exist yet, build it.
• run setup (which will build the pure python sections)
• if the command was build, run the build now.

my implementation of this is shown below. It's designed for only one extension module, and it recompiles all the files every time, so may require further extension to be of more general use. Also note that I've hard coded various unix /s, so if you're porting this to windows make sure you adapt or replace with os.path.sep.

from distutils.core import setup
from distutils.sysconfig import get_python_inc
from Cython.Build import cythonize
import sys, os, shutil

cythonize('delaunay/sphere.pyx')

target = 'build/lib/delaunay/sphere.so'

def build():
  fortran_compiler = 'gfortran'
  c_compiler = 'gcc'
  architecture = 'x86_64'
  python_h_location = get_python_inc()
  build_temp = 'build/custom_temp'
  global target

  try:
    shutil.rmtree(build_temp)
  except OSError:
    pass

  os.makedirs(build_temp) # if you get an error here, please ensure the build/ ...
  # folder is writable by this user.

  c_files = ['delaunay/sphere.c']
  fortran_files = ['delaunay/stripack.f90']

  c_compile_commands = []

  for cf in c_files:
    # use the path (sans /s), without the extension, as the object file name:
    components = os.path.split(cf)
    name = components[0].replace('/', '') + '.'.join(components[1].split('.')[:-1])
    c_compile_commands.append(
      c_compiler + ' -arch ' + architecture + ' -I' + python_h_location + ' -o ' +
      build_temp + '/' + name + '.o -c ' + cf
    )

  fortran_compile_commands = []

  for ff in fortran_files:
    # prefix with f in case of name collisions with c files:
    components = os.path.split(ff)
    name = components[0].replace('/', '') + 'f' + '.'.join(components[1].split('.')[:-1])
    fortran_compile_commands.append(
      fortran_compiler + ' -arch ' + architecture + ' -o ' + build_temp + 
      '/' + name + '.o -c ' + ff
    )

  commands = c_compile_commands + fortran_compile_commands + [
    fortran_compiler + ' -arch ' + architecture + 
    ' -bundle -undefined dynamic_lookup ' + build_temp + '/*.o -o ' + target
  ]

  for c in commands:
    os.system(c)


if 'install' in sys.argv and not os.path.exists(target):
  try:
    os.makedirs('build/lib/delaunay')
  except OSError:
    # we don't care if the containing folder already exists.
    pass
  build()

setup(
  name="delaunay",
  version="0.1",
  ...
  packages=["delaunay"]
)

if 'build' in sys.argv:
  build()

This could be wrapped up into a new Extension class I guess, with it's own build_ext command - an exercise for the advanced student ;)

Answer 2

Simply build and install your vintage Fortran library outside of Python, then link to it in distutils. Your question indicates that you do not intend to temper with this library, so a once-and-for-all install will probably do (using the library's build and installation instructions). Then link the Python extension to the installed external library:

ext_modules = [
    Extension("delaunay",
              sources = ["delaunay/__init__.pyx"],
              libraries = ["delaunay"])
]

This approach is also safe for the case that you realize that you need wrappers for other languages as well, such as Matlab, Octave, IDL, ...

Update

At some point, if you end up with more than a few such external libraries that you want to wrap, it is advantageous to add a top-level build system that installs all these libraries, and manages the building of all the wrappers as well. I have cmake for this purpose, which is great at handling system-wide builds and installations. However, it cannot build Python stuff out of the box, but it can be taught easily to call "python setup.py install" in each subdirectory python, thus invoking distutils. So the overall build process looks like this:

mkdir build
cd build
cmake ..
make
make install
make python
(make octave)
(make matlab)

It is very important to always separate core library code from wrappers for specific front-end languages (also for your own projects!), since they tend to change rather fast. What happens otherwise can be seen at the example of numpy: Instead of writing one general-purpose C library libndarray.so and creating thin wrappers for Python, there are Python C API calls everywhere in the sources. This is what is now holding back Pypy as a serious alternative to CPython, since in order to get numpy they have to support every last bit of the CPython API, which they can't do, since they have a just-in-time compiler and a different garbage collector. This means we are missing out on a lot of potential improvements.

Bottom line:

1. Build general purpose Fortran/C libraries separately and install them system-wide.

2. Have a separate build step for the wrappers, which should be kept as lightweight as possible, so that it's easy to adapt for the next big language X that comes about. If there is one safe assumption, it is that X will support linking with C libraries.