Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Does Python's multiprocessing.Pool support remote subprocesses? [duplicate]

I have recently discovered how awesome Pool.map() is for simple multiprocessing tasks, but my local machine only has two processors and I was hoping to take advantage of some remote computing resources.

Can a processing pool be constructed to use remote machines as well as the local machines in a transparent way?

What python code would I need to run on the remote machines, How would I set up the pool on the local (master) machine?

On the documentation page for multiprocessing, It says Managers are to be used for shared objects, but it's unclear to me how they should be used for a remote Pool. Nothing other than Managers seems make any reference to ports and addresses, so I'm guessing all remote activity happens through this class.

like image 908
Nathan Avatar asked Nov 04 '22 23:11

Nathan


1 Answers

multiprocessing.Pool seems not designed for distributed processing -- if that's what you want to do, I suggest you check out the execnet package instead. That works over SSH pipes and sends code along to the remote Python interpreter if necessary.

like image 144
Fred Foo Avatar answered Nov 11 '22 03:11

Fred Foo