Shared value in parallel python

Question

I'm using ParallelPython to develop a performance-critical script. I'd like to share one value between the 8 processes running on the system. Please excuse the trivial example but this illustrates my question.

def findMin(listOfElements):
    for el in listOfElements:
        if el < min:
             min = el

import pp
min = 0
myList = range(100000)
job_server = pp.Server()
f1 = job_server.submit(findMin, myList[0:25000])
f2 = job_server.submit(findMin, myList[25000:50000])
f3 = job_server.submit(findMin, myList[50000:75000]) 
f4 = job_server.submit(findMin, myList[75000:100000]) 

The pp docs don't seem to describe a way to share data across processes. Is it possible?

If so, is there a standard locking mechanism (like in the threading module) to confirm that only one update is done at a time?

l = Lock()
if(el < min):
     l.acquire
     if(el < min):
         min = el
     l.release

I understand I could keep a local min and compare the 4 in the main thread once returned, but by sharing the value I can do some better pruning of my BFS binary tree and potentially save a lot of loop iterations.

Thanks-

Jonathan

Answer 1

Actually, there is an example at http://www.parallelpython.com/content/view/17/31/#CALLBACK and they simply use the locks from the thread module.

Like JudoWill pointed out, make sure to experiment with how often you should sync the global min in your jobs. If you do it every time you may end up close to serializing your whole calculation.

Answer 2

Parallel Python runs the sub-functions on different processes, so there is no shared memory which means you should not be using a shared value. The callback example mentioned by clackle takes the results of each function and combines them in a callback function which is operating in the original process. To use it properly you should do something similar; in the example given you would calculate local minimums and use a callback function to find the minimum of all the subresults. Hopefully in your real case you can do something similar.