Monitoring tool for distributed micro-services

Question

So I have a set of distributed micro-services, for eg. Elasticsearch, MongoDB, Postgres, a Flask application, an Apache Kafka cluster etc. I want to build a monitoring tool where I can have a dashboard for all kinds of stats around them. For eg, run-time, CPU %, number of processes et al. These and the obvious current status which would tell me if the service is running or not.

Never made a tool like this before, so could do with some help. Is there something already made for this? If not, what are the good practices for making such a thing?

Answer 1

Look at Sensu. Sensu is a pretty advanced monitoring tool. Think of it like distributed nagios. So you install a Sensu agent on each machine which has a large no. Of plugins available for each of the processes you mentioned above. You can also write your own plugins in a language of your choice as long as they are valid executables. Bash, perl, Python, ruby anything would work. Agents publish data to a rabbitmQ, where on data is read by the Sensu server. For dashboards you can simply install uchiwa, which is probably the best Sensu dashboard out there. This allows you to do stuff beyond just monitoring because Sensu server requests updates from all Sensu agents via rabbitmQ. So you can use it like your own event trigger on these machines. Like you could remotely start and shutdown processes too.

Another noteworthy tool is Riemann, http://riemann.io/.

While Sensu is more about sending data via agents, Riemann is all about stream processing at receiving end. So you could have Sensu agents and a Riemann receiver. With Riemann you can build really advanced rules of metric alerts.

Then there's Icinga. Which is also advanced nagios but not as comprehensive as Sensu is.

Lastly if you don't want to go through the process of setting up and maintaining all this, you can just use a hosted service like datadog. But there you would pay for every host you monitor.