.NET API tracing with OpenTelemetry

Question

I am sure someone has already had this problem, but unfortunately not posted anywhere...

I have a simple .NET Web API and using OpenTelemetry for tracing (https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/examples/AspNetCore/Program.cs).

The full registration looks as follows:

services.AddOpenTelemetry()
    .ConfigureResource(resourceBuilder => resourceBuilder.AddService(
        serviceName: "my-web-api",
        serviceVersion: Assembly.GetEntryAssembly()?.GetCustomAttribute<AssemblyInformationalVersionAttribute>()?.InformationalVersion,
        serviceInstanceId: Environment.MachineName))
          .WithTracing(y =>
           {
                y.AddAspNetCoreInstrumentation();

                y.AddOtlpExporter(x =>
                {
                    x.Endpoint = new Uri(Configuration.GetValue<string>("OtlpExporter:Endpoint"));
                });
           });

After I set it up, everything works just fine and I can see traces appearing in Tempo. But as soon the API gets called by another API. No traces get recorded (as if the request did not happen). The root cause of this is that the calling API is adding a header called traceparent with a random GUID like value. Such incoming requests are just being skipped for some reason.

I am using the latest library versions as of now:

OpenTelemetry.Exporter.Console 1.5.1
OpenTelemetry.Exporter.OpenTelemetryProtocol 1.5.1
OpenTelemetry.Extensions.Hosting 1.5.1"
OpenTelemetry.Instrumentation.AspNetCore 1.5.0-beta.1

Again, this seems like something pretty trivial, but not that obvious to locate the real cause of this.

Answer 1

By default, the OpenTelemetry .NET SDK uses the sampling configuration ParentBased(root=AlwaysOn). Which means, if a trace is to be continued (via a traceparent header on an incoming request), the sampling decision encoded in the traceparent header is used. If a new trace is started, it will be always sampled. In your case, this encoded sampling decision is most likely set to not sample most of the time.

In your case, if you want to see all traces of the downstream service, independent of the upstream samplign decision, add this configuration:

.SetSampler<AlwaysOnSampler>()

If you want to see a certain percentage of traces of the downstream service, independent of the upstream samplign decision, add this configuration:

.SetSampler(new TraceIdRatioBasedSampler(0.25))

For most production cases, you probably want to use a ParentBasedSampler combined with a TraceIdRatioBasedSampler.