numpy ctypes "dynamic module does not define init function" error if not recompiled each time

Question

sorry for yet an other question about dynamic module does not define init function. I did go through older questions but I didn't find one which adress my case specifically enought.

I have a C++ library which should export several functions to python ( like ~5 functions defined in extern "C" {} block ). It works just fine when I recompile the library each time I import it. However, if I import it without recompilation it gives error ImportError: dynamic module does not define init function (initmylib)

The very simplified example which reproduce the same behaviour (error) looks like this:

C++ library code in file mylib.cpp

#include <math.h>
// there are some internal C++ functions and classes 
// which are not exported, but used in exported functions
extern "C"{
// one of the functions which should be accessible from python
void oscilator( double dt, int n, double * abuff, double * bbuff ){
    double a = abuff[0];
    double b = bbuff[0];
    for (int i=1; i<n; i++){
        a = a - b*dt;
        b = b + a*dt;
        abuff[i] = a;
        bbuff[i] = b;
    }
}
// there are also other functions but let's keep this example simple
// int initmylib( ){ return 0; } // some junk ... if this makes ctypes happy ?
}

python warper mylib.py of C++ library mylib.cpp :

import numpy as np
from   ctypes import c_int, c_double
import ctypes
import os

name='mylib'
ext='.so'  # if compited on linux .so on windows .dll

def recompile( 
        LFLAGS="",
        #FFLAGS="-Og -g -Wall"
        FFLAGS="-std=c++11 -O3 -ffast-math -ftree-vectorize"
    ):
    import os
    print " ===== COMPILATION OF : "+name+".cpp"
    print  os.getcwd()
    os.system("g++ "+FFLAGS+" -c -fPIC "+name+".cpp -o "+name+".o"+LFLAGS)
    os.system("g++ "+FFLAGS+" -shared -Wl,-soname,"+name+ext+" -o "+name+ext+" "+name+".o"+LFLAGS)

# this will recompile the library if somebody delete it
if not os.path.exists("./"+name+ext ):
    recompile()

lib    = ctypes.CDLL("./"+name+ext )

array1d = np.ctypeslib.ndpointer(dtype=np.double, ndim=1, flags='CONTIGUOUS')

# void oscilator( double dt, int n, double * abuff, double * bbuff )
lib.oscilator.argtypes = [ c_double, c_int, array1d, array1d ]
lib.oscilator.restype  = None
def oscilator( dt, a, b ):
    n  = len(a)
    lib.oscilator( dt, n, a, b )

python program test.py which imports mylib

import os
from pylab import *
from basUtils import * 

# this will delete all compiled files of the library to force recompilation
def makeclean( ):
    [ os.remove(f) for f in os.listdir(".") if f.endswith(".so") ]
    [ os.remove(f) for f in os.listdir(".") if f.endswith(".o") ]
    [ os.remove(f) for f in os.listdir(".") if f.endswith(".pyc") ]

# if I do makeclean() every time it works, if I do not it does not
#makeclean( )

import mylib

a=zeros(100)
b=zeros(100)
a[0] = 1

mylib.oscilator( 0.1, a, b )

plot( a )
plot( b )

show()

I was also trying to make ctypes happy by adding some int initmylib( ){ return 0; } function into mylib.cpp as you can see in the code above. however this will produce error SystemError: dynamic module not initialized properly

I don't have this issue when I compile example of cos_doubles from scipy lecture notes. However, this example works only If I want to import just one function with the same name as the name of the library. I want something more general.

Answer 1

Try running import imp; print imp.find_module('mylib')[1]. Are you surprised that it picks mylib.so instead of mylib.py? The interpreter expects mylib.so to be an extension module, which for CPython 2.x should define an initialization function named initmylib. To avoid accidentally trying to import the shared library, either change the name to something like _mylib.so or mylib.1.0.so -- or just save the file in a directory that's not in sys.path.

Note that Windows extension modules are DLLs, but with a .pyd extension instead of .dll. So import mylib won't try to load mylib.dll.