How grafana dashboard filter Zipkin (tracing, Explore) metrics?

Question

I'd like to ask about zipkin with grafana dashboard. I installed zipkin with grafana using official docker-compose (but storage is elasticsearch. because when I use mysql, an access error happens. es doesn't happen) https://github.com/openzipkin-attic/docker-zipkin#elasticsearch

By the way, I am using zipkin for collecting graphQL execution time with nodejs application. (using opentelemetry library such as HttpInstrumentation, ExpressInstrumentation and GraphQLInstrumentation) And I'd like to show which graphQL queries are slow on some graphical chart.

I try to use prometheus metrics explorers below with the grafana dashboard.

zipkin_collector_messages_total
zipkin_collector_bytes_total
zipkin_collector_spans_total
zipkin_collector_spans_dropped_total
zipkin_collector_messages_dropped_total

But it looks like they can't filter any metrics tags on grafana dashboard ... (because in terms of above controller, we can only use 'instance, job, transport' labels)

Could we do that by prometheus metrics explorers? Or, for filtering by metrics, should we use data storage (such as elastic search, mysql, etc) on grafana dashboard? Thank you so much.

Answer 1

You need to make distinction between traces and metrics. Your saved metrics in the Prometheus are only aggregated data from traces - and it looks like they don't have required dimension for you.

Configure your OTEL collector properly (use spanmetricsprocessor processor - that still unstable processor! - customize dimension config for your need - e.g. graphl.query - sorry I really don't know your tags, so it is blind guess) and export metric to the prometheus. Then you will have in the prometheus also metrics with usefull dimension for your use case and you can filter/group on that.

But I would simply use Zipkin UI directly and filter by min duration (+ tags eventually) to find traces/queries with the longest duration/latency.