I have recently discovered how awesome Pool.map()
is for simple multiprocessing tasks, but my local machine only has two processors and I was hoping to take advantage of some remote computing resources.
Can a processing pool be constructed to use remote machines as well as the local machines in a transparent way?
What python code would I need to run on the remote machines, How would I set up the pool on the local (master) machine?
On the documentation page for multiprocessing, It says Managers are to be used for shared objects, but it's unclear to me how they should be used for a remote Pool. Nothing other than Managers seems make any reference to ports and addresses, so I'm guessing all remote activity happens through this class.
multiprocessing.Pool
seems not designed for distributed processing -- if that's what you want to do, I suggest you check out the execnet
package instead. That works over SSH pipes and sends code along to the remote Python interpreter if necessary.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With