What are java libraries for collecting/storing/presenting application metrics/statistics?

Question

I have a Java application that needs to collect and report a large number of metrics/operational measurements/statistics (different words for different people - I'll just use metrics) about various entities. Metrics can be one of:

1. Some measured numerical value at a particular point in time (number of users currently logged onto the system, % utilization of some resource)
2. Some measured numerical value over a given time interval (number of messages transmitted in the last 5mins)
3. Some enumeration value at a particular point in time (Health status of some entity is ONLINE/WARNING/FAILED/OFFLINE)

'Entities' can be physical devices (a computer, a switch, a modem) but are most often logical entities (Communication channel 4567, User 123456, interface to service A). Additionally, some of these logical entities are static in that they are known ahead of time and don't change during the life of the application, and others may be transient (communcation channel 4567 may only exist for some time, and then is deleted).

I'm looking for libraries to help solve 2 problems:

1. Make it easier to collect and store these bits of information on the server.
2. Present the data to a user in a useful fashion.

Note/Requirements:

1. The server application producing the data is basically a Java process running as a daemon (ie not a webapp)
2. There will be 2 different types of clients.
   1. One is a Java Swing desktop application where most of the 'heavy' use will be. Rich live charting and historical querying abilities are important here.
   2. The other is a web UI. My requirements are not 100% clear here yet, but the needs of this client will be considerably lighter than the swing client. Open standards are needed here (AJAX/HTML not Flash)
3. Can be deployed embedded in proprietary software (ie, no GPL License). Commercial or free.

In the past, my company has attempted a couple of 'home-grown' solutions for this need. Basically the data is shoved into a database in some inconsistent fashion, and the client pulls the data out and is able to chart a single value over a fixed period of time using either home grown or 3rd party charting packages. Frankly, all of our solutions have sucked. Some of the problems I hope a 3rd party solution could fix would be:

1. Encapsulate the code used to update/record the metric so that it doesn't polute the class that is actually measuring the value.
2. Help manage the naming/identifying of the metrics. There can easily be 1000s of them; coming up with some kind of naming scheme myself to ensure uniqueness is a PITA.
3. Deal with historical data nicely. Being able to configure how long data is kept and either archiving or purging old records.
4. On the client side, built-in charting and querying ability, with the ability to transform some of the raw data from the database before being presented to the user.

Sorry for the long description. Thanks for your help.

Answer 1

I can recommend this Java library by Yammer: https://github.com/dropwizard/metrics It offers a nice API to have counters, timings, histograms etc. And it can automatically collect interesting data from servlets by instrumenting them and from the JVM (memory etc).

We store the data then in Graphite: http://graphite.wikidot.com/

Graphite stores the data in a fixed sized database similar to RRD, and can generate charts and dashboards.

Answer 2

RRD4J

Sounds like RRD4J (Apache license) is exactly what you're looking for.
It is a Java implementation of the well established RRDtool. It is built to collect data over time in a Round Robin Database (RRD). Old data is typically archived in RRA archives where the time scale is less fine grained, leading to more space efficient storage.

While I have no direct experience with RRD4J, we have successfully been using RRDTool and Cacti for many years, for health monitoring in our data center. Since RRD is the industry standard when it comes to monitoring, I would definitely recommend you take a look at RRD4J.

Answer 3

You can try Stagemonitor https://github.com/stagemonitor/stagemonitor

It can be paired with elasticsearch/graphite/influxdb for timeseries data storage.

and you can use Grafana to visualise metrics and it can connect to many different databases like elasticsearch, influxdb, prometheus, mysql.