How to write safe/correct multi-threaded code in .NET?

Question

Today I had to fix some older VB.NET 1.0 code which is using threads. The problem was with updating UI elements from the worker thread instead of the UI-thread. It took me some time to find out that I can use assertions with InvokeRequired to find the problem.

Besides the above mentioned concurrent modification problem, there are deadlocks, race conditions, etc. one could run into. As debugging/fixing threading problems is a pain, I'm wondering how I could reduce coding errors/faults in this area and how I could easier find any of them. So, what I'm asking for, is:

• Are there any good patterns to follow when writing multi-threading code? What are the Dos and Don'ts?
• What techniques do you use to debug threading problems?

Please provide some example code if applicable and possible. The answers should be related to the .NET framework (any version).

Answer 1

This could be a massive list - read Joe Duffy's excellent "Concurrent Programming On Windows" for much more detail. This is pretty much a brain dump...

• Try to avoid calling into significant chunks of code while you own a lock
• Avoid locking on references which code outside the class might also lock on
• If you ever need to acquire more than one lock at a time, always acquire those locks in the same order
• Where reasonable, use immutable types - they can be shared freely between threads
• Other than immutable types, try to avoid the need to share data between threads
• Avoid trying to make your types threadsafe; most types don't need to be, and usually the code which needs to share data will need to control the locking itself
• In a WinForms app:
  • Don't perform any long-running or blocking operations on the UI thread
  • Don't touch the UI from any thread other than the UI thread. (Use BackgroundWorker, Control.Invoke/BeginInvoke)
• Avoid thread-local variables (aka thread-statics) where possible - they can lead to unexpected behaviour, particularly on ASP.NET where a request may be served by different threads (search for "thread agility" and ASP.NET)
• Don't try to be clever. Lock-free concurrent code is hugely difficult to get right.
• Document the threading model (and thread safety) of your types
• Monitor.Wait should almost always be used in conjunction with some sort of check, in a while loop (i.e. while (I can't proceed) Monitor.Wait(monitor))
• Consider the difference between Monitor.Pulse and Monitor.PulseAll carefully every time you use one of them.
• Inserting Thread.Sleep to make a problem go away is never a real fix.
• Have a look at "Parallel Extensions" and the "Coordination and Concurrency Runtime" as ways of making concurrency simpler. Parallel Extensions is going to be part of .NET 4.0.

In terms of debugging, I don't have very much advice. Using Thread.Sleep to boost the chances of seeing race conditions and deadlocks can work, but you've got to have quite a reasonable understanding of what's wrong before you know where to put it. Logging is very handy, but don't forget that the code goes into a sort of quantum state - observing it via logging is almost bound to change its behaviour!

Answer 2

I'm not sure how well this will help for the particular application you're working with, but here are two approaches borrowed from functional programming for writing multithreaded code:

Immutable objects

If you need to share state between threads, the state should be immutable. If one thread needs to make a change to the object, it creates a brand new version of the object with the change instead of mutating the object's state.

Immutability does not inherently limit the kind of code you can write, nor is it inefficient. There are lots of implementations of immutable stacks, a variety of immutable trees that form the basis of maps and sets, and other kinds of immutable data structures, and many (if not all) immutable data structures are just as efficient as their mutable counterparts.

Since objects are immutable, its not possible for one thread to mutate shared state under your nose. This means you don't need to acquire locks to write multithreaded code. This approach eliminates a whole class of errors related to deadlocking, livelocking, and raceconditions.

Erlang-style message passing

You don't need to learn the language, but have a look at Erlang to see how it approaches concurrency. Erlang apps can scale pretty much indefinitely because each process is completely seperate from all the others (note: these are not exactly processes, but not exactly threads either).

Processes fire up and simply spin a loop waiting for messages: messages are recieved in the form of tuples, which the process can then pattern match against to see if the message is meaningful. Processes can send other messages, but they are indifferent to whoever recieves the message.

Advantanges to this style is an elimination of locks, when one process fails it doesn't bring down your entire app. Here's a nice summary of Erlang-style concurrency: http://www.defmacro.org/ramblings/concurrency.html

Answer 3

Use FIFOs. Lots of them. It's the ancient secret of the hardware programmer, and it's saved my bacon more than once.

Answer 4

It seems nobody answered the question how to debug multithreaded programs. This is a real challenge, because if there is a bug,it needs to be investigated in real time, which is nearly impossible with most tools like Visual Studio. The only practical solution is to write traces, although the tracing itself should:

1. not add any delay
2. not use any locking
3. be multithreading safe
4. trace what happened in the correct sequence.

This sounds like an impossible task, but it can be easily achieved by writing the trace into memory. In C#, it would look something like this:

public const int MaxMessages = 0x100;
string[] messages = new string[MaxMessages];
int messagesIndex = -1;

public void Trace(string message) {
  int thisIndex = Interlocked.Increment(ref messagesIndex);
  messages[thisIndex] = message;
}

The method Trace() is multithreading safe, non blocking and can be called from any thread. On my PC, it takes about 2 microseconds to execute, which should be fast enough.

Add Trace() instructions wherever you think something might go wrong, let the program run, wait until the error happens, stop the trace and then investigate the trace for any errors.

A more detailed description for this approach which also collects thread and timing information, recycles the buffer and outputs the trace nicely you can find at: CodeProject: Debugging multithreaded code in real time 1