Embedding Python in C: Error when attempting to call Python code in a C callback called by Python code

Question

I have some C code that calls a Python function. This Python function accepts an address and uses WINFUNCTYPE to eventually convert it to a function that Python can call. The C function send as a parameter to the Python function will eventually call another Python function. It is at this last step which causes a crash. So in short I go from C -> Python -> C -> Python. The last C -> Python causes a crash. I've been trying to understand the problem, but I have been unable to.

Can someone point out my problem?

C code compiled with Visual Studio 2010 and run with the args "c:\...\crash.py" and "func1":

#include <stdlib.h>
#include <stdio.h>

#include <Python.h>

PyObject* py_lib_mod_dict; //borrowed

void __stdcall cfunc1()
{
    PyObject* py_func;
    PyObject* py_ret;
    int size;
    PyGILState_STATE gil_state;

    gil_state = PyGILState_Ensure();
    printf("Hello from cfunc1!\n");

    size = PyDict_Size(py_lib_mod_dict);
    printf("The dictionary has %d items!\n", size);
    printf("Calling with GetItemString\n");
    py_func = PyDict_GetItemString(py_lib_mod_dict, "func2"); //fails here when cfunc1 is called via callback... will not even go to the next line!
    printf("Done with GetItemString\n");
    py_ret = PyObject_CallFunction(py_func, 0);

    if (py_ret)
    {
        printf("PyObject_CallFunction from cfunc1 was successful!\n");
        Py_DECREF(py_ret);
    }
    else
        printf("PyObject_CallFunction from cfunc1 failed!\n");

    printf("Goodbye from cfunc1!\n");
    PyGILState_Release(gil_state);
}

int wmain(int argc, wchar_t** argv)
{
    PyObject* py_imp_str;
    PyObject* py_imp_handle;
    PyObject* py_imp_dict; //borrowed
    PyObject* py_imp_load_source; //borrowed
    PyObject* py_dir; //stolen
    PyObject* py_lib_name; //stolen
    PyObject* py_args_tuple;
    PyObject* py_lib_mod;
    PyObject* py_func;
    PyObject* py_ret;

    Py_Initialize();

    //import our python script
    py_dir = PyUnicode_FromWideChar(argv[1], wcslen(argv[1]));
    py_imp_str = PyString_FromString("imp");
    py_imp_handle = PyImport_Import(py_imp_str);
    py_imp_dict = PyModule_GetDict(py_imp_handle); //borrowed
    py_imp_load_source = PyDict_GetItemString(py_imp_dict, "load_source"); //borrowed
    py_lib_name = PyUnicode_FromWideChar(argv[2], wcslen(argv[2]));

    py_args_tuple = PyTuple_New(2);
    PyTuple_SetItem(py_args_tuple, 0, py_lib_name); //stolen
    PyTuple_SetItem(py_args_tuple, 1, py_dir); //stolen

    py_lib_mod = PyObject_CallObject(py_imp_load_source, py_args_tuple);
    py_lib_mod_dict = PyModule_GetDict(py_lib_mod); //borrowed

    printf("Calling cfunc1 from main!\n");
    cfunc1();

    py_func = PyDict_GetItem(py_lib_mod_dict, py_lib_name);
    py_ret = PyObject_CallFunction(py_func, "(I)", &cfunc1);

    if (py_ret)
    {
        printf("PyObject_CallFunction from wmain was successful!\n");
        Py_DECREF(py_ret);
    }
    else
        printf("PyObject_CallFunction from wmain failed!\n");

    Py_DECREF(py_imp_str);
    Py_DECREF(py_imp_handle);
    Py_DECREF(py_args_tuple);
    Py_DECREF(py_lib_mod);

    Py_Finalize();

    fflush(stderr);
    fflush(stdout);
    return 0;
}

Python code:

from ctypes import *

def func1(cb):
    print "Hello from func1!"
    cb_proto = WINFUNCTYPE(None)
    print "C callback: " + hex(cb)
    call_me = cb_proto(cb)
    print "Calling callback from func1."
    call_me()
    print "Goodbye from func1!"

def func2():
    print "Hello and goodbye from func2!"

Output:

Calling cfunc1 from main!
Hello from cfunc1!
The dictionary has 88 items!
Calling with GetItemString
Done with GetItemString
Hello and goodbye from func2!
PyObject_CallFunction from cfunc1 was successful!
Goodbye from cfunc1!
Hello from func1!
C callback: 0x1051000
Calling callback from func1.
Hello from cfunc1!
The dictionary has 88 items!
Calling with GetItemString
PyObject_CallFunction from wmain failed!

I added a PyErr_Print() to the end and this was the result:

Traceback (most recent call last):
  File "C:\Programming\crash.py", line 9, in func1
    call_me()
WindowsError: exception: access violation writing 0x0000000C

EDIT: Fixed a bug that abarnert pointed out. Output is unaffected.
EDIT: Added in the code that resolved the bug (acquiring the GIL lock in cfunc1). Thanks again abarnert.

Answer 1

The problem is this code:

py_func = PyDict_GetItemString(py_lib_mod_dict, "func2"); //fails here when cfunc1 is called via callback... will not even go to the next line!
printf("Done with GetItemString\n");
py_ret = PyObject_CallFunction(py_func, 0);

Py_DECREF(py_func);

As the docs say, PyDict_GetItemString returns a borrowed reference. So, the first time you call here, you borrow the reference, and decref it, causing it to be destroyed. The next time you call, you get back garbage, and try to call it.

So, to fix it, just remove the Py_DECREF(py_func) (or add Py_INCREF(py_func) after the pyfunc = line).

Actually, you will usually get back a special "dead" object, so you can test this pretty easily: put a PyObject_Print(py_func, stdout) after the py_func = line and after the Py_DECREF line, and you'll probably see something like <function func2 at 0x10b9f1230> the first time, <refcnt 0 at 0x10b9f1230> the second and third times (and you won't see the fourth, because it'll crash before you get there).

I don't have a Windows box handy, but changing wmain, wchar_t, PyUnicode_FromWideChar, WINFUNCTYPE, etc. to main, char, PyString_FromString, CFUNCTYPE, etc., I was able to build and run your code, and I get a crash in the same place… and the fix works.

Also… shouldn't you be holding the GIL inside cfunc1? I don't often write code like this, so maybe I'm wrong. And I don't get a crash with the code as-is. Obviously, spawning a thread to run cfunc1 does crash, and PyGILState_Ensure/Release solves that crash… but that doesn't prove you need anything in the single-threaded case. So maybe this isn't relevant… but if you get another crash after fixing the first one (in the threaded case, mine looked like Fatal Python error: PyEval_SaveThread: NULL tstate), look into this.

By the way, if you're new to Python extending and embedding: A huge number of unexplained crashes are, like this one, caused by manual refcounting errors. That's the reason things like boost::python, etc. exist. It's not that it's impossible to get it right with the plain C API, just that it's so easy to get it wrong, and you will have to get used to debugging problems like this.