How do I get visibility / debug high memory usage in a dotnet Core application running on IIS

Question

We have an application we have recent ported to use the Aspnet boilerplate framework and are having a number of issues regarding memory usage. Our initial symptom was intense memory usage with no alleviation in times of high use. Memory use increased over night and over the weekends, but not at the rate it did during peak times. As I type this, I'm monitoring and seeing the app use upwards of 5Gb of memory on the system when a normal usage would be at most 500 - 550mb. We profiled the app using a couple of memory profiling tools and found a couple of potential leaks in framework libraries and in the version of Kestrel we were referencing, but even after repairing the issues, the footprint continued to be heavy. The behavior we see now is the app's memory usage will grow indefinitely but forcing garbage collection in the memory profiling tools will recover large swathes of it. Monitoring the app over the weekend showed that the app was operating normally under light use, but today (Monday) during peak times the app is bleeding memory again. I'm not sure what direction to go in, or how to get visibility on what the actual issue is. The memory profiling tools don't show any obvious leaks or issues in that regard, and the fact that the memory can be reclaimed by forcing garbage collection seems suspicious to me.

Answer 1

Kestrel handles garbage collection (GC) slightly different from ASPNET on IIS. Kestrel has the ability to be much faster than ASPNET and one of the ways it achieves this feat is by reducing GC pressure. Here is an article giving more detail:

https://www.poppastring.com/blog/ASPNETCoreKestrelTheNeedForSpeed.aspx

You may need to rewrite your code taking into consideration what Kestrel GC considers a long-lived object. Without any specifics I can't help much more, but it does sound like your objects are being released by your code and not being collected in a timely manner by the normal GC process.

Answer 2

Unfortunately it looks like the actual issue was buried in the framework we were using, specifically an issue with the dependency injection library not handling transient dependencies correctly. We worked with the third party provider for the library and they included a fix in a more current build, which fixed our issue 100%.