Can Cython code be compiled to a dll so C++ application can call it?

Question

I have a C++ program and it has sort of plugin structure: when program starts up, it's looking for dll in the plugin folder with certain exported function signatures, such as:

void InitPlugin(FuncTable* funcTable);

Then the program will call the function in the dll to initialize and pass function pointers to the dll. From that time on, the dll can talk to the program.

I know Cython let you call C function in Python, but I'm not sure can I write a Cython code and compile it to a dll so my C++ program can initialize with it. An example code would be great.

Answer 1

Using cython-module in a dll is not unlike using a cython-module in an embeded python interpreter.

The first step would be to mark cdef-function which should be used from external C-code with public, for example:

#cyfun.pyx:

#doesn't need python interpreter
cdef public int double_me(int me):
    return 2*me;
        
#needs initialized python interpreter
cdef public void print_me(int me):
    print("I'm", me);

cyfun.c and cyfun.h can be generated with

cython -3 cyfun.pyx

These files will be used for building of the dll.

The dll will need one function to initialize the python interpreter and another to finalize it, which should be called only once before double_me and print_me can be used (Ok, double_me would work also without interpreter, but this is an implementation detail). Note: the initialization/clean-up could be put also in DllMain - see such a version further bellow.

The header-file for the dll would look like following:

//cyfun_dll.h
#ifdef BUILDING_DLL
    #define DLL_PUBLIC __declspec(dllexport) 
#else
    #define DLL_PUBLIC __declspec(dllimport) 
#endif

//return 0 if everything ok
DLL_PUBLIC int cyfun_init();
DLL_PUBLIC void cyfun_finalize();

DLL_PUBLIC int cyfun_double_me(int me);
DLL_PUBLIC void cyfun_print_me(int me);

So there are the necessary init/finalize-functions and the symbols are exported via DLL_PUBLIC (which needs to be done see this SO-post) so it can be used outside of the dll.

The implementation follows in cyfun_dll.c-file:

//cyfun_dll.c
#define BUILDING_DLL
#include "cyfun_dll.h"

#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include "cyfun.h"

DLL_PUBLIC int cyfun_init(){
  int status=PyImport_AppendInittab("cyfun", PyInit_cyfun);
  if(status==-1){
    return -1;//error
  } 
  Py_Initialize();
  PyObject *module = PyImport_ImportModule("cyfun");

  if(module==NULL){
     Py_Finalize();
     return -1;//error
  }
  return 0;   
}


DLL_PUBLIC void cyfun_finalize(){
   Py_Finalize();
}

DLL_PUBLIC int cyfun_double_me(int me){
    return double_me(me);
}

DLL_PUBLIC void cyfun_print_me(int me){
    print_me(me);
}

Noteworthy details:

1. we define BUILDING_DLL so DLL_PUBLIC becomes __declspec(dllexport).
2. we use cyfun.h generated by cython from cyfun.pyx.
3. cyfun_init inizializes python interpreter and imports the built-in module cyfun. The somewhat complicated code is because since Cython-0.29, PEP-489 is default. More information can be found in this SO-post. If the Python-interpreter isn't initialized or if the module cyfun is not imported, the chances are high, that calling the functionality from cyfun.h will end in a segmentation fault.
4. cyfun_double_me just wraps double_me so it becomes visible outside of the dll.

Now we can build the dll!

:: set up tool chain
call "<path_to_vcvarsall>\vcvarsall.bat" x64

:: build cyfun.c generated by cython
cl  /Tccyfun.c /Focyfun.obj /c <other_coptions> -I<path_to_python_include> 

:: build dll-wrapper
cl  /Tccyfun_dll.c /Focyfun_dll.obj /c <other_coptions> -I<path_to_python_include>

:: link both obj-files into a dll
link  cyfun.obj cyfun_dll.obj /OUT:cyfun.dll /IMPLIB:cyfun.lib /DLL <other_loptions> -L<path_to_python_dll>

The dll is now built, but the following details are noteworthy:

1. <other_coptions> and <other_loptions> can vary from installation to installation. An easy way is to see them is to run cythonize some_file.pyx` and to inspect the log.
2. we don't need to pass python-dll, because it will be linked automatically, but we need to set the right library-path.
3. we have the dependency on the python-dll, so later on it must be somewhere where it can be found.

---

Were you go from here depends on your task, we test our dll with a simple main:

//test.c
#include "cyfun_dll.h"

int main(){
   if(0!=cyfun_init()){
      return -1;
   }
   cyfun_print_me(cyfun_double_me(2));
   cyfun_finalize();
   return 0;
}

which can be build via

...
:: build main-program
cl  /Tctest.c /Focytest.obj /c <other_coptions> -I<path_to_python_include>

:: link the exe
link test.obj cyfun.lib /OUT:test_prog.exe <other_loptions> -L<path_to_python_dll>

And now calling test_prog.exe leads to the expected output "I'm 4".

Depending on your installation, following things must be considered:

• test_prog.exe depends on pythonX.Y.dll which should be somewhere in the path so it can be found (the easiest way is to copy it next to the exe)
• The embeded python interpreter needs an installation, see this and/or this SO-posts.

---

IIRC, it is not a great idea to initialize, then to finalize and then to initialize the Python-interpreter again (that might work for some scenarios, but not all , see for example this) - the interpreter should be initialized only once and stay alive until the programs ends.

Thus, it may make sense to put initialization/clean-up code into DllMain (and make cyfun_init() and cyfun_finalize() private), e.g.

BOOL WINAPI DllMain(
    HINSTANCE hinstDLL,  // handle to DLL module
    DWORD fdwReason,     // reason for calling function
    LPVOID lpReserved )  // reserved
{
    // Perform actions based on the reason for calling.
    switch( fdwReason ) 
    { 
        case DLL_PROCESS_ATTACH:
            return cyfun_init()==0;

        case DLL_PROCESS_DETACH:
            cyfun_finalize();
            break;
        case DLL_THREAD_ATTACH:
         // Do thread-specific initialization.
            break;

        case DLL_THREAD_DETACH:
         // Do thread-specific cleanup.
            break;   
    }
    return TRUE;
}

If your C/C++-program already has an initialized Python-interpreter it would make sense to offer a function which only imports the module cyfun and doesn't initialize the python-interpreter. In this case I would define CYTHON_PEP489_MULTI_PHASE_INIT=0, because PyImport_AppendInittab must be called before Py_Initialize, which might be already too late when the dll is loaded.