Cython - sharing Cython modules

Question

I have cython modules base.pyx and derived.pyx. base.pyx and its import definition base.pxd files are defined and is used in derived.pyx file using cimport. The files are placed here.

While using the shared object generated from cython in my Python application, the module is imported correctly, but there is always an error saying ImportError: No module named 'base', when the library is used in Cpp application.

I also added the current path in my Cpp application

  PyObject* path = PySys_GetObject("path");
  PyObject* result = PyObject_CallMethod(path,"append","(s)",".");
  Py_XDECREF(result);

Even then the cython module is not visible to the application. Please let me know what can be the reason for such a path mismatch.

Answer 1

Part of the confusion here is about the behaviour of cimport - OP was trying to use a file derived.pyx that cimported base.pyx and was trying to do that from C++.

cimport does two things:

1. It allows Cython to know of the cdef functions defined in another .pyx or .pxd file, so that it can access those functions. This happens at compile time.

2. It imports that module. This happens at runtime and can be seen by inspecting the generated C code. The import is "important" for two reasons:

   1. The functions used in base.pyx might need global variables or classes to be initialized in base, which is done at import time.

   2. It causes the base shared object to be physically loaded into memory, and initializes some function pointers (in derived) to the cdef functions in base.

   Note that the import doesn't look to actually add base to derived's module dictionary, so it isn't quite the same as a Python import.

I believe part of the thing confusing OP was this slightly unexpected import.

---

The second issue was only revealed on questioning, since it involved a key detail not written in the question. I managed to get the code working as supplied by doing:

python3 setup.py build_ext --inplace
g++ test.cpp -o test `python3-config --libs --includes` ./derived.cpython-36m-x86_64-linux-gnu.so
./test

(where the second line may need altering to match the exact name of the compiled derived module). @PierredeBuyl did something slightly different but similarly found that the code worked without changes.

The problem actually turned out to be that OP was renaming derived.cpython-36m-x86_64-linux-gnu.so and base.cpython-36m-x86_64-linux-gnu.so to libderived.so and libbase.so. This wasn't a problem for derived which was linked directly with the test program but meant that the Python import mechanism couldn't find base (since it was renamed).

---

Posted as community wiki to disassociate myself from as much of the reputation as possible, since I think it was something of a joint effort with a somewhat unsatisfactory solution.