producer/consumer-like multithreading in python

Question

A typical producer-consumer problem is solved in python like below:

from queue import Queue
job_queue = Queue(maxsize=10)    

def manager():
    while i_have_some_job_do:
        job = get_data_from_somewhere()
        job_queue.put(job) #blocks only if queue is currently full

def worker():
    while True:
        data = job_queue.get() # blocks until data available
        #get things done

But I have a variant of producer/consumer problem (not one strictly speaking, so let me call it manager-worker):

The manager puts some job in a Queue, and the worker should keep getting the jobs and doing them. But when the worker get a job, it does not remove the job from the Queue(unlike Queue.get()). And it is the manager which is able to remove a job from the Queue.

So how does the worker get the job while not removing the job from the queue? Maybe get and put is OK?

How does the manager remove a particular job from the queue?

Answer 1

Perhaps your works can't remove jobs completely, but consider letting them move them from the original queue to a different "job done" queue. The move itself should be cheap and fast, and the manager can then process the "job done" queue, removing elements it agrees are done, and moving others back to the worker queue.