How to use a custom OperationId without ending up with memory leaks and it being ignored by Application Insights?

Question

My question is:

What is the correct/better way to inject a custom Correlation/Operation Id in my scenario, which doesn't result in memory leaks and is correctly used and displayed on the Azure Portal?

And here are the specific details of my scenario:

I have a Azure Worker Role that is constantly dequeueing messages from an Azure queue and doing processing. The message contains a Correlation Id (Operation Id) and Operation Name, which I want to be used for all telemetry collected and pushed through Application Insights during the processing work for that message.

The way this is currently implemented is through a custom ITelemetryInitializer, which looks something like this:

public class MiTelemetryInitialiser : ITelemetryInitializer
{
    private readonly ILoggingContext _context;

    public MiTelemetryInitialiser(ILoggingContext context)
    {
        _context = context;
    }

    public void Initialize(ITelemetry telemetry)
    {
        if (string.IsNullOrEmpty(telemetry.Context.Operation.Id) && _context != null)
        {
            telemetry.Context.Operation.Id = _context.OperationId;

            if (!String.IsNullOrWhiteSpace(_context.OperationName))
                telemetry.Context.Operation.Name = _context.OperationName;
        }
    }
}

Then my worker role processing loop look something like this (si:

while(!cancellationToken.IsCancellationRequested) {
    // 1. Get message from the queue
    // 2. Extract operation name and Correlation Id from the message
    TelemetryConfiguration config = TelemetryConfiguration.CreateDefault();

    var loggingContext = //new logging context with operation name and correlation id
    config.TelemetryInitializers.Add(new MiTelemetryInitialiser(loggingContext));
    TelemetryClient telemetryClient = new TelemetryClient(config);        

    // do work

    client.Flush();
}

I am using this TelemetryChannel:

<TelemetryChannel Type="Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.ServerTelemetryChannel, Microsoft.AI.ServerTelemetryChannel"/>

There are two issues:

1. Memory leak (significant) - I am guessing due to the way the ServerTelemetryChannel is implemented (unfortunately unlike most of the .NET SDK - this channel is not open source) and due to the tight loop?
2. Even though client.TrackRequest() is pushing my custom Correlation Id (verified using Fiddler) - Application Insights is overwriting it on the Azure Portal and is displaying its own Operation Id rather than mine. Other telemetry during the request has the correct OperationId which I have set.

Here is a photo of the memory leak after running the loop for ~10 minutes (grows to 1GB+ after a while):

Answer 1

For the memory leak problem - we found the root cause and we will fix it in the next version. However in your code example we recommend to not create a new channel every time and instead reuse the same channel. It will allow better batching and less memory overhead:

TelemetryConfiguration config = TelemetryConfiguration.CreateDefault();

var loggingContext = //new logging context with operation name and correlation id
config.TelemetryInitializers.Add(new MiTelemetryInitialiser(loggingContext));

while(!cancellationToken.IsCancellationRequested) {
    // 1. Get message from the queue
    // 2. Extract operation name and Correlation Id from the message
    TelemetryClient telemetryClient = new TelemetryClient(config);        

    // do work

    client.Flush();
}

Or just use this:

while(!cancellationToken.IsCancellationRequested) {
    // 1. Get message from the queue
    // 2. Extract operation name and Correlation Id from the message
    TelemetryClient telemetryClient = new TelemetryClient();        

    // do work

    client.Flush();
}

and add your telemetry initializer into ApplicationInsigths.config:

<Add Type="Namespace.MiTelemetryInitialiser , AssemblyName" />

See also my blog post on how you can add telemetry initializer to the global singleton configuration:

Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration.Active.TelemetryInitializers.Add(new MiTelemetryInitialiser());

Answer 2

For the operation id problem - In recent versions, specifically for request telemetry, to avoid confusion between RequestTelemetry.ID and TelemetryContext.Operation.ID (from UI perspective), we initialize TelemetryContext.Operation.ID and assign the same to the RequestTelemetry.ID as soon as the request telemetry is created. See example at github. The best way to address this problem of assigning custom operation id is to forcibly assign your operation id, removing string.IsNullOrEmpty check.