Menu
Is there a Python module that would enable me to place instances of non-trivial user classes into shared memory?
By that I mean allocating directly in shared memory as opposed to pickling into and out of it.
multiprocessing.Value and multiprocessing.Array wouldn't work for my use case as they only seem to support primitive types and arrays thereof.
The only thing I've found so far is POSH, but it hasn't changed in eight years. This suggests that it's either super-stable or is out of date. Before I invest time in trying to get it work, I'd like to know if there are alternatives I haven't discovered yet.
I only need this to work on Linux.
662
Shared memory can be a very efficient way of handling data in a program that uses concurrency. Python's mmap uses shared memory to efficiently share large amounts of data between multiple Python processes, threads, and tasks that are happening concurrently.
Python uses a garbage collection algorithm (called Garbage Collector) that keeps the Heap memory clean and removes objects that are not needed anymore. You don't need to mess with the Heap, but it is better to understand how Python manages the Heap since most of your data is stored in this section of the memory.
In CPython, which is what most people use when they use python , all Python objects are represented by a C struct, PyObject .
Open (or create) a shared-memory region. Close a shared-memory region. Map a shared-memory region into a process's address space. Unmap a shared-memory region from a process's address space.
That's kind of a tough one. The best solution I can think of is pickling your objects and using a c_char_p with multiprocessing.sharedctypes. You'd still have to make sure no null bytes got into the c_char_p, either by escaping them or converting to hex.
On second thought, maybe you should go with POSH.
124
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
