synchronization across multiple processes in python

Question

I have a python application that spawns a separate process to do some work (I ran into performance issues using threads due to the GIL (global interpreter lock)). Now what's my methods in python to synchronize shared resources across processes?

I move data into a queue and a spawn process does the work when as it receives data from that queue. But I need to be able to guarantee that the data is coming out in an orderly fashion, same order as it was copied in so I need to guarantee that only one process at any time can read/write from/to the queue. How do I do that best?

Thanks, Ron

Answer 1

I think you need a Semaphore, check this example code:

import threading
import datetime


class ThreadClass(threading.Thread):
    def run(self):
        now = datetime.datetime.now()
        pool.acquire()
        print "%s says hello, World! at time: %s"  % (self.getName(),now)
        pool.release()


pool = threading.BoundedSemaphore(value=1)


for i in range(10):
        t = ThreadClass()
        t.start()

Has this output:

Thread-1 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-2 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-3 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-4 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-5 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-6 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-7 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-8 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-9 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-10 says hello, World! at time: 2013-05-20 18:57:47.609000

Where as:

import threading
import datetime


class ThreadClass(threading.Thread):
    def run(self):
        now = datetime.datetime.now()
        print "%s says hello, World! at time: %s"  % (self.getName(),now)




for i in range(10):
        t = ThreadClass()
        t.start()

Has this output:

Thread-1 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-2 says hello, World! at time: 2013-05-20 18:58:05.
531000

 Thread-4 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-3 says hello, World! at time: 2013-05-20 18:58:05
.531000

 Thread-6 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-5 says hello, World! at time: 2013-05-20 18:58:05
.531000

 Thread-8 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-7 says hello, World! at time: 2013-05-20 18:58:05
.531000

 Thread-10 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-9 says hello, World! at time: 2013-05-20 18:58:0
5.531000

So with a BoundedSemaphore in python you can make sure that before anyone writes to your queue they have to have the semaphore. This doesn't ensure your results are added to the queue in the correct order though.

EDIT:

If you going to do this and keep order your going to need something like this:

import multiprocessing
import datetime
import random
import time

def funfun(number):
    time.sleep(random.randint(0,10))
    now = datetime.datetime.now()
    return "%s says hello, World! at time: %s"  % (number,now)

if __name__ == "__main__":
    pool = multiprocessing.Pool(10)
    for item in pool.imap(funfun,[i for i in range(10)]):
        print item

which prints:

0 says hello, World! at time: 2013-05-21 00:38:48.546000
1 says hello, World! at time: 2013-05-21 00:38:55.562000
2 says hello, World! at time: 2013-05-21 00:38:47.562000
3 says hello, World! at time: 2013-05-21 00:38:51.578000
4 says hello, World! at time: 2013-05-21 00:38:50.578000
5 says hello, World! at time: 2013-05-21 00:38:48.593000
6 says hello, World! at time: 2013-05-21 00:38:52.593000
7 says hello, World! at time: 2013-05-21 00:38:48.593000
8 says hello, World! at time: 2013-05-21 00:38:50.593000
9 says hello, World! at time: 2013-05-21 00:38:51.609000

So with this you could just append to the queue in the right order, and the jobs will wait for their turn to add to the queue.