C code embedded python callback function

Question

C code embedded python callback function，and put data to python queue through callback, when i get data from queue, it's very slow.

Example:

c code like this

static int wrap_func(const int a, const unsigned char *b)
{
  long ret;
  PyObject *arglist;
  PyObject * result = NULL;

  arglist = Py_BuildValue("(s#)", b, a);
  result = PyEval_CallObject(my_callback, arglist);

  /* evaluate result or handle exception */
  ret = PyInt_AsLong(result);
  if (result == NULL)
    return -1;
  Py_DECREF(result);

  return ret;
}

void produce_data()
{
    while(1){
        //produce data to buffer, len is buffer length
        //call callback func 
        wrap_func(buffer, len);
    }
}

compile this c code to so like mywrap.so, and import this so in python python code like this:

import multiprocessing
import mywarp   # mywrap.so

class WorkerThread_a(threading.Thread):
    def __init__(self, workQueue):
        threading.Thread.__init__(self)
        self.workQueue = workQueue
        self.setDaemon(True)
    def run(self):
        while 1:
            try:
                recvdata = self.workQueue.get(block=False)
            except Queue.Empty:
                continue
            #do sth use recvdata

workQueue = multiprocessing.Queue()

def callback_func(a):
    if a:
        workQueue.put(a)
    return 0

def main():
    tmp = WorkerThread_a(workQueue)
    tmp.start()
    mywarp.set_callback(callback_func)
    mywarp.decode_audio()

main()

In python thread, i get data from queue, but i get data very slowly, but in c so, produce data and put to queue through python callback func quickly.

how can i get data from queue quickly like in pure python code.

Answer 1

I think what's happening is that your C code is never releasing the global interpreter lock (GIL), so your Python code never has a chance to run. When you're running multiple threads in Python code, they automatically swap the GIL between them and so share time equally, but this doesn't happen without your intervention in C code.

It should probably work a lot better if you acquire and release the GIL once per loop in your C-code (even though you don't do anything that doesn't need it). All I've really done is added the macros Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS at the start of the function.

static int wrap_func(const int a, const unsigned char *b)
{
  Py_BEGIN_ALLOW_THREADS /* ADDED! */
  /* This should give the chance for the main Python thread
  to run, even though no code goes here */
  Py_END_ALLOW_THREADS /* ADDED */

  long ret;
  PyObject *arglist;
  PyObject * result = NULL;

  arglist = Py_BuildValue("(s#)", b, a);
  result = PyEval_CallObject(my_callback, arglist);

  /* evaluate result or handle exception */
  ret = PyInt_AsLong(result);

  /* This is chang */
  if (result == NULL)
    return -1;
  Py_DECREF(result);


  return ret;
}

(I should say - this is an untested guess that I'm 90% sure is right, but I've been wrong before!)