A process serving application pool 'Classic .NET AppPool' suffered a fatal communication error with the Windows Process Activation Service

Question

I am running an ASP.NET 3.5 web application on IIS 7 on Windows Server 2008.

I received the following error earlier this afternoon:

HTTP Error 503. The service is unavailable.

I looked in the Event logs and discovered that:

A process serving application pool 'Classic .NET AppPool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '3328'. The data field contains the error number.

Here is the full details of that event:

Log Name:      System Source:        Microsoft-Windows-WAS Date:          18/09/2009 14:58:31 Event ID:      5011 Task Category: None Level:         Warning Keywords:      Classic User:          N/A Computer:      computername Description: A process serving application pool 'Classic .NET AppPool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '3328'. The data field contains the error number. Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">   <System>     <Provider Name="Microsoft-Windows-WAS" Guid="{5 .. 0}" EventSourceName="WAS" />     <EventID Qualifiers="32768">5011</EventID>     <Version>0</Version>     <Level>3</Level>     <Task>0</Task>     <Opcode>0</Opcode>     <Keywords>0x80000000000000</Keywords>     <TimeCreated SystemTime="2009-09-18T13:58:31.000Z" />     <EventRecordID>23552</EventRecordID>     <Correlation />     <Execution ProcessID="0" ThreadID="0" />     <Channel>System</Channel>     <Computer>computername</Computer>     <Security />   </System>   <EventData>     <Data Name="AppPoolID">Classic .NET AppPool</Data>     <Data Name="ProcessID">3328</Data>     <Binary>6D000780</Binary>   </EventData> </Event> 

This resulted in:

Application pool 'Classic .NET AppPool' is being automatically disabled due to a series of failures in the process(es) serving that application pool.

Here is the full details of that event:

Log Name:      System Source:        Microsoft-Windows-WAS Date:          18/09/2009 15:03:05 Event ID:      5002 Task Category: None Level:         Error Keywords:      Classic User:          N/A Computer:      computername Description: Application pool 'Classic .NET AppPool' is being automatically disabled due to a series of failures in the process(es) serving that application pool. Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">   <System>     <Provider Name="Microsoft-Windows-WAS" Guid="{5 .. 0}" EventSourceName="WAS" />     <EventID Qualifiers="49152">5002</EventID>     <Version>0</Version>     <Level>2</Level>     <Task>0</Task>     <Opcode>0</Opcode>     <Keywords>0x80000000000000</Keywords>     <TimeCreated SystemTime="2009-09-18T14:03:05.000Z" />     <EventRecordID>23557</EventRecordID>     <Correlation />     <Execution ProcessID="0" ThreadID="0" />     <Channel>System</Channel>     <Computer>computername</Computer>     <Security />   </System>   <EventData>     <Data Name="AppPoolID">Classic .NET AppPool</Data>     <Binary>     </Binary>   </EventData> </Event> 

Concluding in the: HTTP Error 503. The service is unavailable.

Would someone please help me to identify how this happened and what I need to do to prevent it from happening in the future.

Answer 1

This one is a pretty nasty one. I recommend getting the IIS Debug Diagnostics tools. You can probably alleviate the symptom by turning off "Rapid Fail Protection" in the properties of your application pool, but that will just mask the problem. What is going on is that your application is experiencing an exception so often/so repeatedly that the service shuts itself off. You'll need to run the diagnostic tool to capture everything going in and out to find the exception that's occuring.

This can be a very laborious task because the tool will gather gigabytes of junk to sift through. If you're lucky, the turning off of the "Rapid Fail Protection" might allow the exception to proceed through to the browser, but my guess is that the server will just end up hanging.

Answer 2

If you are using NServiceBus make sure whatever account the application pool is running under has permissions to read off of the message queue.

NServiceBus runs a background task on application startup that timesout after 10 seconds, causing the w3wp process to be killed if it can't read from the queue. After n (default of 5) w3wp crashes Rapid-Fail Protection kicks in and stops the application pool.

http://blog.jonathanoliver.com/2010/11/nservicebus-w3wp-exe-processes-crashing/