Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is python multiprocessing Queue safe for object put?

When I put an object in Queue, Is it necessary to create deep copy of object and then put in Queue?

like image 844
Majid Avatar asked Jan 16 '17 09:01

Majid


1 Answers

If you can ensure that the Object is only processed in one Thread, this is not a problem. But if you can't, it is recommended to use a deep copy.

The Queue object doesn't do this autmatically if you put the object into it.

See Refs

  • Multithreading, Python and passed arguments
  • Python in Practice: Create Better Programs Using Concurrency... p.154

Keep in mind that the object need to be able to be pickled (Multiprocessing Basics)

It usually more useful to be able to spawn a process with arguments to tell it what work to do. Unlike with threading, to pass arguments to a multiprocessing Process the argument must be able to be serialized using pickle. This example passes each worker a number so the output is a little more interesting.

like image 121
ppasler Avatar answered Sep 23 '22 09:09

ppasler