Trace-level processor filtering

Question

I have Health Probes probing my backend which is traced with auto-instrumentation for NodeJS. This causes a LOT of noise in my Logz.io (where the traces end up).

I tried solving this with a filter processor:

processors:
  filter/health_probes:
    error_mode: ignore
    traces:
      span:
        - 'attributes["http.user_agent"] == "Edge Health Probe"'

This removes the topmost/parent span (HEAD /) but all the child spans that follow the parent HEAD / span are still retained.

Is there a way to do trace-level filtering based on the parent/topmost span of a trace?

Answer 1

The best way to do this is to introduce head sampling at the system receiving the health check probes to not sample those requests. That should propagate to all children.

An alternative is to use tail sampling in the collector, which requires that all spans for a given trace flow through the same collector instance and that they're held for some period of time to allow the collector to have a "complete" view of the trace. It is significantly more complicated for anything but an example setup.