Multithreaded Single Producer Multiple Consumer Implementation

Question

I would like to implement a Multiple file downloading with pattern of single producer and multiple consumer.

What I have: - Code which finds new links to be downloaded in a loop - When a new link is found - it calls download function - Download function accepts source file path and destination file path and downloads the file.

What I want to do - I want to download X number of files simultaneously (I dont know total number of files) - At any times I should be able to download X files simultaneously - as soon as 1 of the X file finish downloading - the calling function should be able to add new download right away - which in turn downloading right away

• So I have a producer function which keeps adding new download to queue (at any time maximum X downloads)
• Multiple X thread which consumes the downloads and start downloading individually. Once it finishes download - the producer should be able to add new download - which will spawn new thread.

EXAMPLE would be really appreciated

Answer 1

For this P/C problem all you need is a BlockingCollection<T>.

//shared and thread-safe
static BlockingCollection<string> queue = new BlockingCollection<string>(100);

// Producer
queue.Add(fileName);  // will block when full

// Consumer
if (queue.TryTake(out fileName, timeOut))  // waits when empty
  ...

You'll want to fine-tune it a little with timeouts and CancellationTokens.