How to call a dll function in parallel in Python

Question

I have a python program that uses ctypes to call a dll function to which I pass a pointer. It is supposed to write data continuously to that pointer and I want my program to loop and read the pointers contents. A rough template would look like:

from ctypes import *
import copy
lib = cdll.lib
pointer = c_char_p(" "*100) #however large a buffer I need
#thread this
lib.dostuff(pointer)
#end thread
while True:
    data = pointer.value
    print data

dostuff(), in my specific case, is written in C and opens a file and decodes it, running the data as a stream into a character array.

The problem is that I can't use the regular threading module in python since the thread holds the GIL either since reading the dll is considered file I/O or because the dll itself does file I/0. Therefore, the loop does not run until dostuff() is complete. What is the reason it blocks (will dll calls always block?) and how can I get around this?

EDIT: ----------SOLVED---------------------- As samplebias points out below, ctypes releases the GIL　lock. I discovered that the blocking issue in my program was that I was running a queue: The code looked a bit like this

import Queue
from threading import Thread

queue = Queue()

def runthread():
     lib.dostuff(pointer)
     while True:
        queue.put(pointer.value)

thread = Thread(target=runthread)
thread.start()
while True:
    data = queue.get()
    dostuffwithdata(data)

The program was blocking because queue.get() blocks when the queue is empty, until something goes inside! Of course, since I didn't thread the dll call alone, it finished before I pushed the pointers results to the queue. The solution looks a bit like this:

import Queue
from threading import Thread

queue = Queue()

def runthread():
     q = Thread(target=lib.dostuff, args=(pointer,))
     q.start()
     while True:
         queue.put(pointer.value)

thread = Thread(target=runthread)
thread.start()
while True:
   data = queue.get()
   dostuffwithdata(data)

I hope this helps someone!

Answer 1

This can definitely work using threading, since ctypes releases the GIL before calling a C function. This enables (among other things) the C routine to call back into Python code without creating a deadlock.

About the only problem you'll encounter is how to signal to the DLL to stop delivering data, but there are ways to solve that as well, e.g. passing a 2nd pointer as a flag indicating when to return, for example.

Here is a working example along the lines of your question, e.g. the GIL is released and the Python and C code run concurrently:

Shared object: test.c

#include <stdint.h>
#include <stdio.h>

void
dostuff(uint64_t *ptr)
{
    while (1)
        (*ptr)++;
}

Compile it:

% gcc -shared -g -o test.so test.c -fPIC

Python code: test.py

import ctypes
import sys
import time
import threading

lib = ctypes.cdll.LoadLibrary('./test.so')
val = ctypes.c_uint64(0)

def loop():
    lib.dostuff(ctypes.byref(val))

t1 = threading.Thread(target=loop)
t1.start()

for i in range(1000):
    sys.stdout.write('%s ' % val.value)
    sys.stdout.flush()
    time.sleep(0.05)

Output

% python test.py 
0 24664442 48388062 71628820 94834416 118004961 141095893 164936784 ... ...