Adding multiple alerts on single grafana panel

Question

I have configured grafana dashboard to monitor promethus metrics for some of the spring boot services. I have a single panel and a prom query for every service on it.

Now I want to add alerts for each on of those queries. But I couldn't find a way to add multiple alerts on single panel. I could add only only for one of the queries.

Is there a way to do it? Or would I need to split panel into multiple panels?

Answer 1

You can specify the query that the alert threshold is evaluating within the 'conditions' but it will still be just one alert. As such your Alert message won't include anything to distinguish which specific query condition triggered the alert, it's just whatever text is in the box (AFAIK there's not currently any way to add variables to the alert message).

I've ended up with a separate dashboard which isn't generally viewed, just for alerts with multiple panels for each alert. You can quickly duplicate them by using the panel json and a search/replace for the node name, service name etc.

Answer 2

As Andrew Poodle mentioned, you can create multiple queries in that single panel and use the "multiple conditions" feature in the alert section. They will be under one alert.

However, you can use the grafana notification templating in order to distinguish your alert conditions as seen in the pic below.