Menu
I have a Parallel.ForEach() async loop with which I download some webpages. My bandwidth is limited so I can download only x pages per time but Parallel.ForEach executes whole list of desired webpages.
Is there a way to limit thread number or any other limiter while running Parallel.ForEach?
Demo code:
Parallel.ForEach(listOfWebpages, webpage => { Download(webpage); }); The real task has nothing to do with webpages, so creative web crawling solutions won't help.
815
We can restrict the number of concurrent threads created during the execution of a parallel loop by using the MaxDegreeOfParallelism property of ParallelOptions class.
maxConcurrency decides how many threads can execute parallelly at the same time, Let's say if it is set to 2 and total records to be processed are 10 then remaining 8 will be queued. For Parallel Foreach, By default (when no maxConcurrency provided), all routes run in parallel.
You don't have to do anything special, Parallel. Foreach() will wait until all its branched tasks are complete. From the calling thread you can treat it as a single synchronous statement and for instance wrap it inside a try/catch.
No, it doesn't block and returns control immediately. The items to run in parallel are done on background threads.
You can specify a MaxDegreeOfParallelism in a ParallelOptions parameter:
Parallel.ForEach( listOfWebpages, new ParallelOptions { MaxDegreeOfParallelism = 4 }, webpage => { Download(webpage); } ); MSDN: Parallel.ForEach
MSDN: ParallelOptions.MaxDegreeOfParallelism
172
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
