Sporadic windows service failure under .NET 4, followed by port blocked on restart attempt

Question

About once a day, I am receiving the following error in our mission-critical trading service.

Source: .NET Runtime, Type: Error, Application: Application.exe, Framework Version: v4.0.30319, Description: The process was terminated due to an internal error in the .NET Runtime at IP 000006447F281DBD (000006447F100000) with exit code 80131506.

After receiving this error and trying to restart the application, it appears that the sockets we were bound to have not been cleaned up from the previous (failed) execution because we receive a System.ServiceModel.AddressAlreadyInUseException when trying to Bind the socket during startup.

I have two questions around this.

1. We need to understand why the first error is occuring, do you have any information from the error codes, etc.
2. We need a way to be able to Bind successfully after the error has occured. Any suggestions on how to cleanup the ports during the next startup.

The environment the application is running under is

• Microsoft Windows Server 2003 R2
• Standard x64 Edition
• Service Pack 2
• 2x 4Core Intel CPU X5365 @ 3.00GHz
• 16.0 GB of RAM.

Answer 1

This is the ExecutionEngineException from the earlier .NET days. You cannot catch it in .NET 4.0, AppDomain.UnhandledException won't run.

The generic diagnostic for this exception is that the integrity of the garbage collected heap was compromised. A typical trigger is unmanaged code writing past the end of a buffer. Or it can be environmental, virus scanners have a knack to cause this problem. Especially Symantec security products. Which is somewhat likely in your case, given that the ports aren't being closed automatically when your service terminates. It is also technically possible for a bug in the CLR to cause this.

I'd thus recommend:

• Inspect your source code base and thoroughly review any unmanaged code that's used.
• Contact vendors of 3rd party components and ask about known heap corruption problems.
• Review the configuration of the machine on which this code runs. Disable add-ons where possible, temporarily disable anything that isn't strictly necessary to run your service
• Retarget your project to the .NET 3.5 SP1 framework.