Design pattern to monitor distributed system?

Question

I have a distributed system: 12-14 applications running on 10 boxes (each with about 8 cores). My apps are heavily multithreaded.

In the course of the day my apps are pretty busy. Latency is critical for what I do.

Given the circumstances, I have a new additional requirement that I have to monitor a bunch of in memory objects spread across these apps and generate some reports (could be a web page or a text file doesn't matter).

I am looking for design patterns relevant to the monitoring work. What is bothering me is that I should not be introducing any latency by way of some monitoring/observer threads doing anything nasty. If it helps, I am mostly C++ at this point so low level stuff like shared memory etc are definitely on the table.

Answer 1

Yours is a very broad question!

Here are some starting ideas:

• Event driven architecture allows you to invert the message flow and makes it easier to have asynchronous workflows.

• EDA also works nicely with Event Sourcing strategies for state management.

• Message queues are typically well suited as a transport mechanism for events and, well, messages. They normally adhere to some specified set of performance characteristics, but you have to see if they are adequate for your purposes.

• If you need even more speed, you can use a lock-free structure like a ring buffer as an in-memory queue to decouple the main business logic from the reporting logic.

I realize my answer is very generic, but hopefully it will be of help.