Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

PLINQ: how to run a ParallelQuery on more than 4 threads?

Tags:

c#

plinq

Update - changed the title of the question to reflect what I'm really after

Consider the following piece of code:

// this query generates 12 instances of Func<int>, which each when executed
// print something to the console and wait for 1 second.
var actions = Enumerable.Range(0, 12).Select(i => new Func<int>(() =>
{
    Console.WriteLine("{0} - waiting 1 sec", i);
    Thread.Sleep(1000);
    return 1;
}));

// define a parallel query. Note the WithDegreeOfParallelism call here.
var query = from action in actions.AsParallel().WithDegreeOfParallelism(12)
            select action();

// execute, measuring total duration
var stopw = Stopwatch.StartNew();
query.ToList();
Console.WriteLine(stopw.Elapsed);
Console.WriteLine(Environment.ProcessorCount); // 3 on my machine

When omitting the call to WithDegreeOfParallelism, this executes in 4 chunks, taking about 4 seconds in total, which is what I would expect since my CPU count is 3.

However, when calling WithDegreeOfParallelism with any number above 4, I always get 3 chunks, and the total duration does not go under 3 seconds. I would expect that a value of 12 would get a total duration of (a little more than) 1 second.

What am I missing? And how can I enforce the parallel execution of more than 4 non-CPU intensive tasks, which is what I'm after?

Update: I could of course go back to manually spinning up threads, but I was hoping that the new PFX library would make this a bit easier... Anyway, the code below gives me about 1 second total execution time

List<Thread> threads = new List<Thread>();
for (int i = 0; i < 12; i++)
{
    int i1 = i;
    threads.Add(new Thread(() =>
    {
        Console.WriteLine(i1);
        Thread.Sleep(1000);
    }));
}
threads.ForEach(t => t.Start());
threads.ForEach(t => t.Join());
like image 555
jeroenh Avatar asked Jul 27 '11 11:07

jeroenh


2 Answers

Try starting new tasks in your parallel loop with the option TaskCreationOptions.LongRunning. They will start right away, instead of waiting until a thread on the threadpool becomes available.

like image 166
Gebb Avatar answered Oct 01 '22 09:10

Gebb


As I said WithDegreeOfParallelism is setting only a upper bound.Try increasing your tasks from 10 to 100. You will ended up with around 10 seonds for all 100 of them. Your code is good for larger number of tasks having smaller operations. and add Console.WriteLine("{0} threads " ,Process.GetCurrentProcess().Threads.Count); inside your task then you can see how many threads are created.( Thread count is not the count of plinq created threads. See how it increasing).

There are lots of ways to do the parallelism with PLinq . Read this article http://msdn.microsoft.com/en-us/library/dd997411.aspx. You need to choose best way for the relevant requirement to get better performance.

like image 33
Jayantha Lal Sirisena Avatar answered Oct 01 '22 09:10

Jayantha Lal Sirisena