Passing python objects across interpreters

Question

I would like to exchange objects between two python interpreters working on the same (and different ) machines.

Is there a mechanism to achieve the same?

~Vijay

To be more specific: I have few processes running on same (and different) machine(s). Few of these processes produce objects which are consumed by the other processes. I am trying to tackle a producer/consumer problem among these processes.

I was previously running these processes on the same python interpreter as threads, and was able to use queues to solve the producer-consumer problem. However due to increasing loads I have decided to shift some processes to other machines. Now these processes residing on different machines(and interpreters) need to share the access to a queue. I would like to know if there is a mechanism which helps me do this efficiently.

I could write a single class which manages the queue and handles the objects to/from the other processes using sockets.

I think this is a common problem if one wants to use python to handle large amounts of data. Hence I was interested to know if there were other more elegant solutions available.

~Vijay

Answer 1

Generally, you need to serialize the object. In Python that means pickle and unpickle. You can pickle an object on one side, transfer it somehow (socket, etc.) the other and unpickle it. Use the pickle or cPickle modules for that.

The Pyro package is a complete package that does this in a transparent way.