How to design multithreaded application

Question

I have a multithreaded application. Each module is executed in a separate thread. Modules are:

- network module - used to receive/send data from network
- parser module - encode/decode network data to internal presentation
- 2 application module - perform some application logic on the above data one after other
- counter module - used to gather statistics from other modules
- timer module - used to schedule timers
- and much more ...

All threads using message queues for inter thread communication (std::deque sync by conditional variable and mutex).

Some modules are used by others ones (e.g. all modules use timer and counter) and this for each message received from network wich should be handled in very high rates.

This is pretty complex application and the design looks "reasonable". From other hand, I'm not sure that such design, thread per module, is the "best" one? In particular, I'm afraid that such design "encorage" a lot of context switches.

What do you think?

Is there're any good guidelines or open source project to learn from how to do "correct" design of threaded application?

Answer 1

Thread-per-function designs are just naive: they assume that by separating tasks - by module - onto threads, that some kind of scalability will be achieved.

This kind of design is inefficient, as very few task breakdowns yield exactly as many tasks as there are CPUs.

Far more rational designs are to break tasks down into 'jobs' - and then use thread pooling mechanisms to dispatch those jobs. Advantages over the thread-per-module approach:

• Thread pools take advantage of all cores. with thread-per-module if you have modules < cores you have cores sitting idle.

• Thread pools minimize contention and resources by maintaining a parity between active threads, and cores. with thread-per-module, if modules > cores you incur needless extra context switches and (on some platforms) each thread exhausts other limited per process resources (like virtual memory).

• Thread pools let a "module" do multiple jobs at a time. thread-per-module means that the busiest module still only gets one core.

Answer 2

I wouldn't call myself an expert an multi-threaded design. But I've at least worked with threads enough to have run into various issues trying to design them to work together (communication, locking resources, waiting for threads to end, etc).

At this point, my general rule of thumb is that I must justify the existence of each new thread. For example, if the network layer I'm using provides both a synchronous and an asynchronous API, can I really justify making the network code use synchronous calls in a new thread instead of just using the asynchronous calls in the main thread? In your case, how many modules actually need a thread of their own for a specific reason. Are there any that could instead just be called in turn from the main thread?

If some threads have no good reason for existing, then you might be able to save yourself some trouble and complexity by just putting that module in the main thread.

Now of course, there are good justifiable reasons for putting things in threads. Such as making synchronous calls that may block for a long time, keeping a GUI thread responsive while performing a long task, or being able to take advantage of parallel processing of a large task on a multi-core system.

I don't know of any particular "correct" way to do it. A lot of it really comes down to the details of what your application is actually supposed to do.

Answer 3

A good guideline is to put operations that might block (such as I/O) in its own thread. Your network module is a definite candidate here. Have your network thread use select (I assume UNIX here) to block on input.

Asynchronous events are good in separate threads as well. Your timer module looks like a good candidate here.

You might want to put your other modules in one thread to decrease complexity of your application. BUT, you might want to split them up if you have a multi-processor system.

Have a good strategy for locking resources and mutex handling to prevent deadlocks. A dependency graph (using a whiteboard!) might help here to get your design correct.

Good luck! Sounds like a complex system which will cause many hours of fun development!