How to Reload a Python3 C extension module?

Question

I wrote a C extension (mycext.c) for Python 3.2. The extension relies on constant data stored in a C header (myconst.h). The header file is generated by a Python script. In the same script, I make use of the recently compiled module. The workflow in the Python3 myscript (not shown completely) is as follows:

configure_C_header_constants() 
write_constants_to_C_header() # write myconst.h
os.system('python3 setup.py install --user') # compile mycext
import mycext
mycext.do_stuff()

This works perfectly fine the in a Python session for the first time. If I repeat the procedure in the same session (for example, in two different testcases of a unittest), the first compiled version of mycext is always (re)loaded.

How do I effectively reload a extension module with the latest compiled version?

Answer 1

You can reload modules in Python 3.x by using the imp.reload() function. (This function used to be a built-in in Python 2.x. Be sure to read the documentation -- there are a few caveats!)

Python's import mechanism will never dlclose() a shared library. Once loaded, the library will stay until the process terminates.

Your options (sorted by decreasing usefulness):

1. Move the module import to a subprocess, and call the subprocess again after recompiling, i.e. you have a Python script do_stuff.py that simply does

   import mycext
   mycext.do_stuff()
   

   and you call this script using

   subprocess.call([sys.executable, "do_stuff.py"])
   

2. Turn the compile-time constants in your header into variables that can be changed from Python, eliminating the need to reload the module.

3. Manually dlclose() the library after deleting all references to the module (a bit fragile since you don't hold all the references yourself).

4. Roll your own import mechanism.

   Here is an example how this can be done. I wrote a minimal Python C extension mini.so, only exporting an integer called version.

   >>> import ctypes
   >>> libdl = ctypes.CDLL("libdl.so")
   >>> libdl.dlclose.argtypes = [ctypes.c_void_p]
   >>> so = ctypes.PyDLL("./mini.so")
   >>> so.PyInit_mini.argtypes = []
   >>> so.PyInit_mini.restype = ctypes.py_object 
   >>> mini = so.PyInit_mini()
   >>> mini.version
   1
   >>> del mini
   >>> libdl.dlclose(so._handle)
   0
   >>> del so
   

   At this point, I incremented the version number in mini.c and recompiled.

   >>> so = ctypes.PyDLL("./mini.so")
   >>> so.PyInit_mini.argtypes = []
   >>> so.PyInit_mini.restype = ctypes.py_object 
   >>> mini = so.PyInit_mini()
   >>> mini.version
   2
   

   You can see that the new version of the module is used.

   For reference and experimenting, here's mini.c:

   #include <Python.h>
   
   static struct PyModuleDef minimodule = {
      PyModuleDef_HEAD_INIT, "mini", NULL, -1, NULL
   };
   
   PyMODINIT_FUNC
   PyInit_mini()
   {
       PyObject *m = PyModule_Create(&minimodule);
       PyModule_AddObject(m, "version", PyLong_FromLong(1));
       return m;
   }