Shared array of objects using multiprocessing in python

Question

I am trying to create an array of shared objects (instances of a class) between different processes in a python program. Each of these objects would be modified in the program. For this purpose I am using multiprocessing as follows:

import multiprocessing
import numpy as np
sh = multiprocessing.RawArray (ctypes.py_object, 10)
f = np.frombuffer(sh, dtype=object)

and the error I get is:

Traceback (most recent call last):
  File "<pyshell#14>", line 1, in <module>
    f = np.frombuffer(sh, dtype=object)
ValueError: cannot create an OBJECT array from memory buffer

Now my question is, first of all, is this generally the right way to tackle this problem, and second, what is my mistake in the above code? Thank you in advance.

Answer 1

ctypes.py_object represents a PyObject * in C. It is a pointer to a struct that represents a Python object, resides in the private memory of the process, and contains more pointers. Another process can't access it; trying to share a pointer between processes is not useful.

See also this answer by Alex Martelli.