Multiprocessor and Performance

Question

I'm facing a really strange problem with a .Net service.

I developed a multithreaded x64 windows service.

I tested this service in a x64 server with 8 cores. The performance was great!

Now I moved the service to a production server (x64 - 32 cores). During the tests I found out the performance is, at least, 10 times worst than in the test server.

I've checked loads of performance counters trying to find some reason for this poor performance, but I couldn't find a point.

Could be a GC problem? Have you ever faced a problem like this?

Thank you in advance! Alexandre

Answer 1

This is a common problem which people are generally unaware of, because very few people have experience on many-CPU machines.

The basic problem is contention.

As the CPU count increases, contention increases in all shared data structures. For low CPU counts, contention is low and the fact you have multiple CPUs improves performance. As the CPU count becomes significantly larger, contention begins to drown out your performance improvements; as the CPU count becomes large, contention actually starts reducing performance below that of a lower number of CPUs.

You are basically facing one of the aspects of the scalability problem.

I'm not sure however where this problem lies; in your data structures, or in the operating systems data structures. The former you can address - lock-free data structures are an excellent, highly scalable approach. The latter is difficult, since it essentially requires avoiding certain OS functionality.

Answer 2

There are way too many variables to know why one machine is slower than the other. 32 core machines are usually more specialized where an eight core could just be a dual proc quad core machine. Are there vm's or other things running at the same time? Usually with that many cores, IO bandwidth becomes the limiting factor (even if the cpu's still have plenty of bandwidth).

To start off, you should probably add lots of timers in your code (or profiling or whatever) to figure out what part of your code is taking up the most time.

Performance troublshooting 101: what is the bottleneck ( where in the code and what subsystem (memory, disk, cpu) )