multiprocessing and ctypes with pointers

Question

I have multiProcessing.Process objects whose target functions take input and output queue.

To the output queue they put some data, that is a wrapped ctypes structure with internal pointers. Of course, the pickle module, that should serialize the data, breaks:

ValueError: ctypes objects containing pointers cannot be pickled

Can I somehow get my ctypes structures with pointers out of my child processes without dumping them to files?

The code is below

# -*- coding: utf-8 -*-
import multiprocessing as mp

from liblinear import *
from liblinearutil import *


def processTarget(inQueue, outQueue):
    while(not inQueue.empty()):
        inVal = inQueue.get()

        #training model
        y, x = [1,-1], [{1:inVal, 3:3*inVal}, {1:-1,3:-1}]
        prob  = problem(y, x)
        param = parameter('-c 4 -B 1')
        m = train(prob, param)


        outQueue.put((inVal * 2, m))
        print "done", inVal
        inQueue.task_done()

def Main():
    processes = []
    inQueue = mp.JoinableQueue()
    for i in xrange(10):
        inQueue.put(i)

    outQueue = mp.JoinableQueue()
    for i in xrange(5):
        process = mp.Process(target=processTarget, args=(inQueue, outQueue))
        print "starting", i
        process.start()
        print "started", i
    inQueue.join()

    print "JOINED"

    while(not outQueue.empty()):
        print outQueue.get()



if __name__ == '__main__':
    Main()

Answer 1

When you use multiprocessing every process has its own address space. The pointer would not be valid in another process.

translate the object to a python object or to a ctypes type without pointers and it should work.

Keep in mind that changes to the object that occur in the other process will not be reflected in the parent unless you send the object back on the queue.