Write TraceId to logs with nlog

Question

I'm trying to setup my net6 project to send traces and logs to grafana loki and tempo for better observability. I'm successfully setup sending traces to tempo using opentelemetry.instrumentation(All traces have TraceId). Now I'm trying to get TraceId for other logs, but can't get it. I'm using NLog.Web.AspNetCore(5.3.2) and NLog.DiagnosticSource(5.0.0) packages and as mentioned here - https://github.com/NLog/NLog.DiagnosticSource - I can get TraceId via ${activity:property=TraceId} variable, but when I use layout like "consoleLayout": "${activity:property=TraceId} [${time} ${uppercase:${level}]} ${logger} ${aspnet-request-method}..." this variable is empty.

I did not find any information that would need to configure something else. As far as I understand, TraceID is present by default, and I only need to get it. What I missed?

Answer 1

I was able to get TraceId in nlog after I have added

builder.Services.AddLogging(o => o
    .Configure(x => x.ActivityTrackingOptions = ActivityTrackingOptions.TraceId | ActivityTrackingOptions.SpanId));

Answer 2

It is likely the TraceId field may contain very big number of unique values across different log lines - an unique value per each processed request with the enabled tracing. Grafana Loki creates an unique log stream per each unique set of log fields (excluding message and timestamp fields). See these docs. This means it will create an unique log stream per each unique TraceId field value. Grafana Loki stores data for every stream in a separate file. It also keeps index data per each stream in memory. This means that Grafana Loki may suffer from high number of small files and high memory usage in the case of millions of unique StreamId values. This is called high cardinality issue.

The solution is to store the TraceId in the message itself, and do not extract it into a separate field. Then you can use line filters for finding messages with the needed TraceId.

P.S. The provided solution isn't the best from usability perspective. You can store StreamId in a dedicated field without worrying about high cardinality issues when using alternative log management systems such as ElasticSearch or VictoriaLogs (I work on it). These systems support fast full-text search over high-cardinality fields. ElasticSearch is in widespread use, while VictoriaLogs requires 30x less RAM and 15x less disk space than ElasticSearch for the same amounts of production logs.