How to debug a 0xc0000409 error in PerfMon.dll? / AssemblyBinding

Question

Problem description

I am trying to debug a problem on a customer's server (Win 2012R2) where one of our .NET Web Applications is hosted in their IIS (version 8.5.9600.16384). Recently, one of our technical supports changed a minor internal setting in the Web.config file of the application and according to them, nothing else was changed or done to the system. The application was not updated either, and had been running without any issues before.

Since that change, the AppPool in which the site is configured cannot be started anymore. According to the System Event logs, there are a few attempted starts, each resulting in a worker process crash, until the Rapid-Fail Protection of the pool stops the startup process entirely.

Also, since I'm on a customer's server, I only have limited debugging options.

Findings

Reported in the System Event log by WAS:

A process serving application pool '[our pool]' suffered a fatal communication error with the Windows Process Activation Service. The process id was '5664'. The data field contains the error number.

Application pool '[our pool]' is being automatically disabled due to a series of failures in the process(es) serving that application pool.

A look at the corresponding Application Error log reveals the following:

Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x52157ba0 Faulting module name: PerfMon.dll, version: 8.0.10977.0, time stamp: 0x59cfb424 Exception code: 0xc0000409 Fault offset: 0x000f8c7b Faulting process id: 0x1620 Faulting application start time: 0x01d423ff75bef49f Faulting application path: C:\Windows\SysWOW64\inetsrv\w3wp.exe Faulting module path: C:\Program Files\Microsoft Monitoring Agent\Agent\APMDOTNETAgent\V8.0.10918.0\PerfMon.dll Report Id: b56327b3-8ff2-11e8-80d5-005056a52299 Faulting package full name: Faulting package-relative application ID:

While googling for this problem I mainly focused on PerfMon.dll issues and the provided error code 0xc0000409, which is apparently either related to a corrupted registry entry or a Stack Buffer Overflow, but in both cases I've simply been unable to figure out what the cause could be.

I also tried the crash dump analysis described here https://blogs.msdn.microsoft.com/parvez/2016/08/06/iis-application-pool-crash-and-debug-diag/

But in the end it seemed like it was just a round-trip back to the original problem, as this was the only error found in the dump:

The assembly instruction at PerfMon!DllGetClassObject+966fb in C:\Program Files\Microsoft Monitoring Agent\Agent\APMDOTNETAgent\V8.0.10918.0\PerfMon.dll from Microsoft Corp. has caused an unknown exception (0xc0000409) on thread 2

Which is basically what I've gotten from the event log already.

My main question here is not only what might be the cause and a potential fix to this problem, but also how to properly find the cause/solution in the first place.

UPDATE

Using Fusion Logs, I was able to find three binding errors for System.Web.DynamicData, System.Web.Extensions and System.ServiceModel.web.

The operation failed. Bind result: hr = 0x80004005. Unspecified error

Assembly manager loaded from: >C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll Running under executable C:\Windows\SysWOW64\inetsrv\w3wp.exe --- A detailed error log follows.

=== Pre-bind state information === LOG: DisplayName = System.ServiceModel.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35 (Fully-specified) LOG: Appbase = [app root path] LOG: Initial PrivatePath = [app root bin directory] LOG: Dynamic Base = C:\Windows\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files\root\bee9e73a LOG: Cache Base = C:\Windows\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files\root\bee9e73a LOG: AppName = b83a4ee Calling assembly : (Unknown).

Rejecting code sharing because a dependent assembly did not match the conditional APTCA share mode

Searching for any information on this I was only able to find an old error related to MVC3/MVC4, which required a configuration change. That is unrelated to this issue, though. This might also mean that the original error has nothing to do with the PerfMon at all and that the original error is just a weird side effect.

Answer 1

Just wanted to add that we had this (or a very similar) issue now on a Windows Server 2012 and resolved it by removing the Microsoft Monitoring Agent APM service as described in this rather brilliant blog post: http://kevingreeneitblog.blogspot.com/2017/03/scom-2016-agent-crashing-legacy-iis.html

(The issue described in the blog post supposedly only affects apps running on older .NET framework versions, but we experienced in with an app pool running only version 4.)

From our event log:

Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x52157ba0
Faulting module name: PerfMon.dll, version: 8.0.11072.0, time stamp: 0x59c01c4c
Exception code: 0xc0000409
Fault offset: 0x000f8c7b
Faulting process id: 0x1ae8
Faulting application start time: 0x01d4eac448929ba3
Faulting application path: C:\Windows\SysWOW64\inetsrv\w3wp.exe
Faulting module path: C:\Program Files\Microsoft Monitoring Agent\Agent\APMDOTNETAgent\V8.0.11072.0\PerfMon.dll
Report Id: 8a700b62-56b7-11e9-8124-001dd8b71cba
Faulting package full name: