Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python: How to share an Unpickable Object

I am doing some multiprocessing and I need to share an instance between two processes. I have been trying to use the multiprocessing module to try and accomplish this but there isn't much hope to share anything that can't be pickled it seems; so far I have tried to use a manager and create a proxy object to handle my object by following this SO question and this SO question. I understand that sharing mutable object instances isn't exactly python's forte, but what is the easiest way to do this?

To layout my situation more clearly, I am working on UNIX systems exclusively so it uses forks and copy-on-write memory management to my understanding. This object that I need to share is read-only on the main process but read and write on the subprocess. The easiest way I can think of doing this is to just share a reference to the object instance in memory to the subprocess and communicate between processes not to write when the object is being used by the main process.

like image 812
user2909415 Avatar asked Feb 11 '26 01:02

user2909415


1 Answers

I solved this problem myself, but for anyone who comes to this thread with the same problem:

I made the mistake of instantiating a SyncManager() object then calling the BaseManager.register() method assuming this would work since SyncManager is just a subclass of BaseManager(). I didn't look into exactly why this is the case, but I do know the fix, and that is to just instantiate a BaseManager() and register your Python object with that instead.

like image 164
user2909415 Avatar answered Feb 12 '26 15:02

user2909415