Is thread time spent in synchronization too high?

Question

Today I profiled one of my C# applications using the Visual Studio 2010 Performance Analyzer. Specifically, I was profiling for "Concurrency" because it seemed as though my app should have more capacity then it was demonstrating. The analysis report showed that the threads were spending ~70-80% of their time in a Synchronization state.

To be honest, I'm not sure what this means. Does this mean that the application is suffering from a live-lock condition?

For context... there are ~30+ long-running threads bound to a single AppDomain (if that matters) and some of the threads are very busy (Ex. while(true) { _waitEvent.WaitOne(0); //do stuff }).

I realize this is a fairly vague question... I guess I'm looking for some elucidation on the meaning the Synchronization state of threads. How much is too much, and why? Is ~75% really bad? Do I have too many threads? or should I just start looking in other areas?

Answer 1

I'm not sure what this means.

It means that the threads were on average spending 75% of their time waiting for another thread to finish some work.

Does this mean that the application is suffering from a live-lock condition?

Maybe!

To clarify for readers unfamiliar with the term: a 'deadlock' is when two threads are both waiting for each other to finish, and therefore they wait forever. A 'live lock' is a situation where two threads are trying to avoid a deadlock, but due to their poor choices, spend most of their time waiting anyway. Imagine for example a table with two people, a fork and a knife. Both people wish to pick up both utensils, use them, and then put them down. Suppose I pick up the knife and you pick up the fork. If we both decide to wait for the other to put the utensil down, we are deadlocked. If we both realize that we're about to deadlock, and I put down the knife, and you put down the fork and then I pick up the fork and you pick up the knife, we are live-locked. We can repeat this process indefinitely; we're both working to resolve the situation, but we're not communicating effectively enough to actually resolve it quickly.

However, my guess is that you're not in a live-lock situation. My guess is rather that you simply have enormous contention on a small number of critical resources that can only be accessed by one thread at a time. Occam's Razor would indicate that you should assume the simple hypothesis -- lots of threads taking turns using a scarce resource -- rather than the complicated hypothesis -- a whole bunch of threads all trying to tell each other "no, you go first".

There are ~30+ long-running threads bound to a single AppDomain (if that matters) and some of the threads are very busy (Ex. while(true) { _waitEvent.WaitOne(0); //do stuff }).

Sounds awful.

I realize this is a fairly vague question.

Yes, it is.

How much is too much, and why?

Well, suppose you were trying to drive across town, and you and every other driver in the city spent 75% of your time stopped at traffic lights waiting for other drivers. You tell me: is that too much, and why? Spending an hour in traffic to drive for 15 minutes distance might be perfectly acceptable to some people and utterly unacceptable to other people. Every time I take SR 520 at rush hour I spend an hour in traffic to move a distance that should take 15 minutes; that wasn't acceptable to me so now I take the bus.

Whether this lousy performance is acceptable to you and your customers or not is your call. Fixing performance problems is expensive. The question you should be asking is how much profit you'll gain by taking on the expense of diagnosing and fixing the problem.

Is ~75% really bad?

Your threads are taking four times longer than they need to. Doesn't seem too good to me.

Do I have too many threads?

You almost certainly do, yes. 30 is a lot.

But that is completely the wrong technical question to ask in your situation. Asking "do I have too many threads?" is like trying to fix traffic congestion by asking "does this city have too many cars?" The right question to ask is "why are there so many traffic lights in this city where there could be freeways?" The problem isn't the threads; the problem is that they are waiting on each other instead of driving on through to their destinations without stopping.

should I just start looking in other areas?

How on earth should we know?

Answer 2

Without really knowing the structure of your program, I can only really tell you what Synchronization means in relation of threads. I can't really tell you what the problem in your program is.

Synchronization basically means that you coordinate so to speak the concurrent running of threads when actions have to be taken on resources that are shared by these threads, in order to avoid corruption of your data.

If you have an string, for example, that your two threads are writing to, then, if you didn't have thread synchronization (ie using AutoResetEvents or Semaphores, etc) then one thread could be in the middle of altering the string in some fashion, be interrupted by the OS (maybe it's time slice was up), and now the second thread may be starts reading from this string, which is in an indeterminate state. This will wreak having with your program, so in order to avoid such things, and many other possible errors that can be caused by threading, you synchronize access to the shared resource, by locking it, so that only one thread at a time can write/read to/from it, and any other thread wanting to do so while the other is doing so, has to wait until our first thread has released the lock it holds.

This is, in a very simplified explanation, what thread synchronization means and what it is for.
There is many more things that come with threading, but that's the subject of an entire book.

As to what your "Synchronization State of Threads" may mean, I would guess it means that a lot of threads spend their time waiting on other threads that are holding some shared resource.

Essentially that means your program is really not working concurrently, but is doing things in serial, as threads are spending their time waiting on other threads. This means that the program is not written very well to really work concurrently, which, by the way is not necessarily an easy feat to achieve, depending on the situation, obviously.

Hope this helps.

Answer 3

There are a few ways that thread synchronization can kill performance:

1. The instructions for implementing synchronization take time to execute, especially for synchronization routines that require a transition to kernel mode such as Thread.Sleep(). In the worst case, a lone thread making frequent calls to synchronization routines introduces a whole lot of overhead for no real benefit.
2. Whenever more than one thread needs exclusive access to a resource at the same time, at least one thread gets stuck waiting. The worst case scenario here is that there is some central resource which everyone needs to access frequently. In this case, multithreaded code is in serious danger of becoming an expensive, slow, and complicated way to have one thread working at a time.

As for how much is too much: Synchronization is something that takes time but doesn't really do any useful work. Therefore, from a performance perspective the ideal amount of synchronization is always zero. Hence the high value that is placed on shared-nothing architectures and immutable data structures. Both are techniques to help organize code in a way that eliminates or reduces the need for synchronization.

Of course the world isn't ideal so some amount of synchronization is usually inescapable. But even then it should be done using the lightest-weight constructs possible. For example, don't use a lock statement when a method in Interlocked will do. Or reduce the frequency with which it needs to happen by designing threads to send work product to a central data structure in batches, rather than making a lot of high-frequency updates.