What are the reasons why the CPU usage doesn’t go 100% with C# and APM?

Question

I have an application which is CPU intensive. When the data is processed on a single thread, the CPU usage goes to 100% for many minutes. So the performance of the application appears to be bound by the CPU. I have multithreaded the logic of the application, which result in an increase of the overall performance. However, the CPU usage hardly goes above 30%-50%. I would expect the CPU (and the many cores) to go to 100% since I process many set of data at the same time.

Below is a simplified example of the logic I use to start the threads. When I run this example, the CPU goes to 100% (on an 8/16 cores machine). However, my application which uses the same pattern doesn’t.

public class DataExecutionContext
{
    public int Counter { get; set; }

    // Arrays of data
}

static void Main(string[] args)
{
    // Load data from the database into the context
    var contexts = new List<DataExecutionContext>(100);
    for (int i = 0; i < 100; i++)
    {
        contexts.Add(new DataExecutionContext());
    }

    // Data loaded. Start to process.
    var latch = new CountdownEvent(contexts.Count);
    var processData = new Action<DataExecutionContext>(c =>
    {
        // The thread doesn't access data from a DB, file, 
        // network, etc. It reads and write data in RAM only 
        // (in its context).
        for (int i = 0; i < 100000000; i++)
            c.Counter++;
    });

    foreach (var context in contexts)
    {
        processData.BeginInvoke(context, new AsyncCallback(ar =>
        {
            latch.Signal();
        }), null);
    }

    latch.Wait();
}

I have reduced the number of locks to the strict minimum (only the latch is locking). The best way I found was to create a context in which a thread can read/write in memory. Contexts are not shared among other threads. The threads can’t access the database, files or network. In other words, I profiled my application and I didn’t find any bottleneck.

Why the CPU usage of my application doesn’t go about 50%? Is it the pattern I use? Should I create my own thread instead of using the .Net thread pool? Is there any gotchas? Is there any tool that you could recommend me to find my issue?

Thanks!

Answer 1

There are many things that could, potentially, cause this behavior.

First, what type of CPU do you have? If you have an i7 or similar processor, the OS will see this as having 8 cores, when in reality, it has 4 cores with 2 hyperthreads/core. For most operations, hyperthreading does not really provide the same scalability as a second core, even though the OS sees it this way. I've had this cause my overall CPU usage to appear lower to the OS...

Second, it's possible you have some form of true sharing occurring. You mention that you have locking - even if it's kept to a minimum, the locks may be preventing you from scheduling this effectively.

Also, right now, you're scheduling all 100 work items, right up front. The os is going to have to page in and out those 100 threads. You may want to restrict this to only allowing a certain number to process at a given time. This is much easier using the new Task Parallel Library (just use Parallel.ForEach with a ParallelOptions setup to have a maximum number of threads) - but can be done on your own.

Given that you're scheduling all 100 items to process simulataneously, the paging may be hampering the ability to get maximum throughput.

Also, if you're doing any other "more real" work - you may be getting false sharing issues, especially if you're working with arrays or collections that are shared (even if the elements you're process are not shared).

I'd recommend running this under the concurrency profiler in VS 2010 - it will give you a much clearer picture of what is happening.