save cython extension by pickle

Question

I have a class or extension type written in cython like this:

cdef class Self_Organized_Map:
     cdef 
     def __cinit__(self,np.ndarray data,.....):
     ....

I created a python module from this cython file (which its name is som.pyx) using distutils , then I import this in python and used it to create and train a model , but when I want to save my model using pickle it gives me this err :

TypeError: can't pickle som.Self_Organized_Map objects

what is wrong with pickle or my code ? does pickle cant save extension object ?

Answer 1

Cython classes aren't pickleable by default, so you need to implement the Pickle interface yourself. There's a number of different levels you can do this on but __getstate__ and __setstate__ is the most user-friendly level so it's a good place to start unless you have good reasons otherwise.

Where the contents of the class are pickleable it's as easy as returning a tuple of them in __getstate__ and the reverse operation in __setstate__. Memoryviews themselves aren't pickleable, but have a base attribute that may be.

cdef class C:
    cdef double[:] array
    cdef python_obj
    cdef int integer

    def __init__(self,array,python_obj,integer):
        self.array = array
        self.python_obj = python_obj
        self.integer = integer

    def __getstate__(self):
        return (self.array.base, # memoryviews aren't pickleable, need to get underlying object
                          # and hope it's pickleable
                self.python_obj, self.integer)

    def __setstate__(self,x):
        self.array, self.python_obj, self.integer = x

If your class holds a C or C++ object then it's much more complicated. For simple types good places to start are just copy the memory into a bytearray or to take advantage of Cython's default struct<->dict interconversion. However if the class contains pointers then this won't work and you need to implement a reliable load/save mechanism for it in C/C++.

Answer 2

Since Cython 0.26 (released July 2017), cdef classes can automatically be pickled, as long as they do not contain pointers or unions. For classes containing structs, automatic pickling can be enabled with the @cython.auto_pickle(True) decorator. It is disabled by default due to high code overhead an other reasons.

More information can be found in the changelog and on the website of Stefan Behnel.