How do I compile a Fortran library for use with Python? (f2py may not be an option)

Question

I'm trying to compile a fortran90 library (specifically this one) in order to call it from python (3.4.0). Generally in this case I would write a wrapper for f2py and call it a day, but the library itself makes use of derived types, which seems to be making f2py fail. The full stderr is pasted here, but the relevent line is

getctype: No C-type found in "{'typename': 'optim_type', 'typespec': 'type'}", assuming void.

The other option, based on the numpy documentation is to use ctypes, which also fails

Python 3.4.0 (default, Jun 19 2015, 14:20:21) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.ctypeslib.load_library('libLBFGS', '/home/kaplane/src/TOOLBOX_OPTIMIZATION_shared/lib')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/kaplane/.local/lib/python3.4/site-packages/numpy/ctypeslib.py", line 123, in load_library
    return ctypes.cdll[libpath]
  File "/usr/lib/python3.4/ctypes/__init__.py", line 426, in __getitem__
    return getattr(self, name)
  File "/usr/lib/python3.4/ctypes/__init__.py", line 421, in __getattr__
    dll = self._dlltype(name)
  File "/usr/lib/python3.4/ctypes/__init__.py", line 351, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /home/kaplane/src/TOOLBOX_OPTIMIZATION_shared/lib/libLBFGS.so: invalid ELF header

What I can't figure out is exactly what is invalid about the ELF header. The output from $ readelf -h is the same (excepting number, size, and locations of program and section headers) as for a shared library that works.

How I'm compiling the library

Compiling on my local machine I use gfortran instead of ifort, and have the compiler flags set as

OPTF =  -O3 -shared -fPIC
OPTC =  -O3 -shared -fPIC
OPTL =  -O3 -shared -fPIC
AR= ar 
ARFUNCT= cruvs

in the Makefile.inc file. I also run a script

find ./ -name "Makefile" | xargs sed -i -e 's/lib\([A-Z]*\)\.a/lib\1.so/g'

so that the libraries are labeled as .so instead of .a. This doesn't seem to affect the operation of the examples programs.

What I'd like to know

I think the best option is to figure out how to compile the library such that I don't get that invalid ELF error. Failing that I'd need to figure out how to compile Fortran modules with derived types, but the searching I've done is less than promising.

Answer 1

f2py is writen for Fortran77 Code and therefore does not support most of the features of Fortran90+, such as derived types, allocatable arrays, etc.

My own workaround incorporated writing a Fortran wrapper routine arround the subroutines I wanted to use. In this wrapper routine I copied all allocatable arrays (because that was the only unsupported feature that was used) to fixed size arrays (f2py also seems to have an undocumented maximum arrays size :/ ). These fixed size arrays, along with the size of the original arrays was then used as output of the fortran wrapper routine.

Additionally I wrote a python wrapper routine for the generated f2py library that read those fixed size arrays (read LARGE), including the size information and only returned the actual data (removing all the unused rows/ columns, etc. from the fixed size array).

This approach was only possible, because I had full control and knowledge of the source files and expected data. I would not recommend it if your work might ot be used by somebody outside of your reach.

---

As an alternative you should have a look at Cython. This provides an almost native way to exchange data between Fortran and Python routines using iso_c_binding [2]. For a minimum working example look here. A great talk about that can also be found in the 1st comment of this question (for reference).

I used the above workaround because I couldn't get it to work back then. But the great talks and tutorials I just mentioned have been added since, that should make it a whole lot easier.