how to control asynchronous access to multiple underlying objects in Python?

Question

How can I efficiently manage exclusive access to several underlying objects for many client threads in python?

At the moment, I have arbitrary numbers of callback threads (ROS Python) sharing access to a single compute object with semaphores like:

def my_callback(self, data):
    self.sem.acquire()
    response = self.compute_object.compute(data)
    self.sem.release()
    self.publish_response(response)

However I want to be able to create arbitrary numbers of instantiations of the underlying compute object and give the callback exclusive access to the first one that becomes available. Is something roughly like this possible with the standard libs?

def my_callback(self, data):
    compute_object_id = self.sem.acquire()
    response = self.compute_objects[compute_object_id].compute(data)
    self.sem.release(compute_object_id)
    self.publish_response(response)

Answer 1

The Queue (Queue.Queue in python2) (queue.Queue in python3) is a nice solution to the asynchronous multi-producer, multi-consumer problem.

Create the queue:

import queue
# asynch queue holding compute objects ready to accept new jobs
self.compute_object_queue = queue.Queue(maxsize=compute_object_instances)

# initiate compute_object N times and add each instance to the queue
for i in range(compute_object_instances):
    self.compute_object_queue.put(compute_object(param1, param2))

Using the queue:

compute_object = self.compute_object_queue.get()
results = compute_object.compute(data)
self.compute_object_queue.put_nowait(compute_object)