Why is Application Insights Performance Counter Collection causing high CPU usage

Question

Logging performed by our Performance Team has indicated that this line specifically, is killing our CPUs

Microsoft.AI.PerfCounterCollector!Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.Implementation.PerformanceCounterUtility.ExpandInstanceName()

One theory was the the regexes used to identify Perf Counters in the library are recursing

https://adtmag.com/blogs/dev-watch/2016/07/stack-overflow-crash.aspx

I've inspected the Perf Counter names and nothing looks particularly out of kilter regarding the names and the regexes should have no trouble chewing over them. Certainly for large periods of time there are no issues whatsoever.

I've now turned on Applications Insights Diagnostic Logging in an attempt to observe the issue (in a test environment)

Has anyone else observed this, how can we mitigate this?
We have ensured DeveloperMode is NOT set to on.

Answer 1

this answer most likely won`t be usable now as AppInsights 2 code is much improved, but in my case it was double call AddApplicationInsightsTelemetry(). It adds collector each time this method is called, and due to lack of synchronization inside perf counter collector code it creates CPU spikes.

So avoid calling multiple times AddApplicationInsightsTelemetry, or use AppInsights 2.x. (and the best thing is to do both).

Answer 2

Do the counters you're collecting utilize instance placeholders in their names? If the instance name is known at build time, getting rid of placeholders may significantly improve performance. For instance, instead of

\Process(??APP_WIN32_PROC??)\% Processor Time

try using

\Process(w3wp)\% Processor Time

Also, how many counters are you collecting overall?