IIS System.OutOfMemoryException issue Web service

Question

We have an Asp.Net Web Service having

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.     at System.Threading.Thread.StartInternal(IPrincipal principal, StackCrawlMark& stackMark)     at System.Threading.Thread.Start()

We are refreshing app pool as a temporary fix. Issues seem re-occurs every 4-5 hours. Web Service basically returns the data from Stored Procedure from SQL server DB. It is a legacy application haven't done any code change in years. Is it possible an application stable for years have any memory leak issue all of sudden?

Answer 1

You're going to need to analyze the IIS application pool's memory dump to trouble shoot this. I've done the same in the past and it isn't too difficult once you get comfortable with the tools.

1. WinDbg - https://go.microsoft.com/fwlink/p/?LinkId=536682 - Make sure you only install the Debugging Tools for Windows component as this links to the full SDK
2. Windows Symbol Server - srvc:\symbolshttps://msdl.microsoft.com/download/symbols. Within WinDbg File menu, select Symbol File Path and add this path (you can change location of c:\symbols but do ensure the directory exists and is local to the system doing the debug work).
3. SOS WinDbg extension - C:\Windows\Microsoft.NET\Framework64\v4.0.30319\SOS.dll. Copy this DLL to the Debugging Tools for Windows Directory

Once you have the tools installed, you need to get a memory dump and analyze it.

• Get a memory dump of your application via the task manager. Make sure to use the same bit task manager as your application, i.e. use a 64-bit instance of task manager if your IIS application pool is running in 64-bit.
• Use the appropriate 32/64-bit version of WinDbg and then open the memory dump via the file menu in WinDbg.
• Once the memory dump is loaded, use this command to load sos: ​.loadby sos clr
• There are a bunch of useful sos commands available for troubleshooting, but the one we're interested in for memory issues would be this one: !dumpheap -stat. This will give you all the currently loaded types and how much memory they are using.
• You will also probably want to confirm it isn't a thread contention issues. Commands that can help with that are: !sos.syncblk to see if any threads are blocked and !runaway​ to see if any threads have been running for an excessive amount of time. Both of these scenarios can cause an application to become unresponsive and would be fixed via an IISRESET or recycle.

Again, there is a bit of a learning curve with WinDbg but once you get over that hump, it is super easy to quickly diagnose issues like these.