How to free memory allocated by external C libraries interfacing with Cython module where the memory is ultimately returned to a Python process?

Question

I am brand new to Cython, but basically I have this application that needs significant performance increases, and so my team and I are trying to rewrite our bottlenecks in Cython and in C.

For the slowest part of our application, I wrote some C code that gets compiled into a library and cdef extern imported into a Cython module, which I believe is a .pyx file. Essentially, the code in the pyx file is basically just a wrapper that returns calls to the C library functions. Finally, there is a Python process (the main application) that imports all of the functions defined in the pyx file and uses these results.

I believe I have a memory leak because in the C code, the results that I need to pass to the Python process are at times dynamically allocated. My issue is that I don't know how to free this memory once the Python process has made use of it.

Example Python Code

from examplecython import *

def foo(data):
    context = data.context
    value = call_pyx_function(context, data)
    return value

def bar(results):
    for data in results:
        res = foo(data)
        do_something_with_res(res)
        # I want to free here

Example Cython Code

cdef extern from "my_lib.h"
    char * my_function(const char * context, int data)

def call_pyx_function(context: bytes, int x):
    return my_function(context, x)

Example C Code

#define BUFSIZE 256

char *
my_function(const char * context, int x) {
    char * retbuf;
    int res;

    retbuf = (char *)malloc(BUFSIZE * sizeof(char));

    res = do_some_math(x, context);

    int length = snprintf(retbuf, BUFSIZE, "%d", res);
    if (length >= BUFSIZE) {
        exit(EXIT_FAILURE);
    }

    return retbuf;
}

If anyone has any suggestions for how and where I can free this memory, that would be very much appreciated.

Answer 1

You can directly import free from libc.stdlib:

from libc.stdlib cimport free

def bar(results):
    for data in results:
        res = foo(data)
        try:
            do_something_with_res(res)
        finally:
            free(res)

(Note you need the try/finally because you want it to be freed even if something throws an exception)

You can make this easier with a context manager or a wrapper that deletes in __del__ / __dealloc__:

@contextlib.contextmanager
def freeing(res):
    try:
        yield res
    finally:
        free(res)

def bar(results):
    for data in results:
        with freeing(foo(data)) as res:
            do_something_with_res(res)

Or (Might get freed much later, probably slower, but (almost) guaranteed to be freed eventually)

# (in pyx file)
cdef class MallocedResource:
    cdef void* res;

    def __init__(self, res):
        # Note: This "steals" res. Don't free `res`
        # as it is freed when this class's storage is freed
        self.res = <void *>res

    def __dealloc__(self):
        free(self.res)

def call_pyx_function(context: bytes, int x):
    return MallocedResouce(my_function(context, x))

# No need to change python code, so you can't forget to use try/finally.