Wait for blocking collection (queue) to decrease in size in C#

Question

I'm working on project with the following workflow:

Part One:

• Event arrives asynchronously and is queued in blocking queue, we'll call that Q1
• Thread picks up next available item from that queue
• Item ends up running {N} number of tasks in parallel
• Each task queues its result on a second queue, we'll call that Q2.
• When processing of item finishes, the next item is read off the queue.

Part Two:

• Another thread reads off of Q2 one object at a time and works on the result

So, the problem here is, every item on the first queue ends up running a large number of tasks in parallel, and each task queues its result. The second queue must be processed serially, one item at a time though, and it's getting flooded.

---

My Question

I need a mechanism that will make the thread processing Q1 wait until the number of items in Q2 is below a specific threshhold. What's the best way to achieve this? Is there any way to have an event driven solution rather than a polling solution?

Answer 1

Instead of using a Queue<T>, you can use a BlockingCollection<T> for Q2. If you set its BoundedCapacity, calls to Q2.Add() will block when the capacity is reached. This will automatically throttle the processing of Q1, as the N tasks will begin blocking if they can't add to the final queue.