C# server scalability issue on linux

Question

I've a C# server developed on both Visual Studio 2010 and Mono Develop 2.8. NET Framework 4.0

It looks like this server behaves much better (in terms of scalability) on Windows than on Linux. I tested the server scalability on native Windows(12 physical cores), and 8 and 12 cores Windows and Ubuntu Virtual Machines using Apache's ab tool.

The windows response time is pretty much flat. It starts picking up when the concurrency level approaches/overcomes the number of cores.

For some reason the linux response times are much worse. They grow pretty much linearly starting from level 5 of concurrency. Also 8 and 12 cores Linux VM behave similarly.

So my question is: why does it perform worse on linux? (and How can I fix that?).

Please take a look at the graph attached, it shows the averaged time to fulfill 75% of the requests as a function of the requests concurrency(the range bar are set at 50% and 100%).

I have a feeling that this might be due to mono's Garbage Collector. I tried playing around with the GC settings but I had no success. Any suggestion?

Some additional background information: the server is based on an HTTP listener that quickly parses the requests and queues them on a thread pool. The thread pool takes care of replying to those requests with some intensive math (computing an answer in ~10secs).

Answer 1

You need to isolate where the problem is first. Start by monitoring your memory usage with HeapShot. If it's not memory, then profile your code to pinpoint the time consuming methods.

This page, Performance Tips: Writing better performing .NET and Mono applications, contains some useful information including using the mono profiler.

Excessive String manipulation and Boxing are often 'hidden' culprits of code that doesn't scale well.

Answer 2

Try the sgen garbage collector (and for that, Mono 2.11.x is recommended). Look at the mono man page for more details.

Answer 3

I don't believe it's because of the GC. AFAIK the GC side effects should be more or less evenly distributed across the threads.

My blind guess is: you can fix it by playing with ThreadPool.SetMinThreads/SetMaxThreads API.