Different objects from putting and getting in queues

Question

I am trying to implement a queue to share some objects between processes in Python (for example a list). However, what I put in the queue is a different object from what I get afterwards:

from multiprocessing import Queue
q = Queue()
a = [1,2,3]
print(id(a)) # prints 4389597128
q.put(a)
b = q.get()
print(id(b)) # prints 4389600080

This does not happen if I use atomic elements, such as numbers.

Why is this the case? How can I put and get the same object into the queue?

Answer 1

You see this behavior because all objects added to a multiprocessing Queue are pickled when inserted, and unpickled when they are retrieved:

Note When an object is put on a queue, the object is pickled and a background thread later flushes the pickled data to an underlying pipe.

This happens even if the consumer from the Queue is running in the same process as the producer. The identity of an object is not preserved through pickling by default (though see this question for a way you can try to preserve it by customizing the pickling process for an object). However, you may be better off adjusting your implementation to not rely on the id of the object remaining the same through the pickling/unpickling process, if possible.

You see the id stay the same for some built-in immutable types (like small integers) because Python interally caches instances of those, and always re-uses the same one. So the unpickled int ends up being a reference to the exact same int that you put into the queue. This isn't the case for custom types. If you try it with a large number (which Python does not cache), you'll see the id changes after you pull it from the Queue, same as it does with custom types.