Using custom performance counters across appdomain recycles

Question

I have an ASP.NET application which tracks statistics by creating and writing to custom performance counters. Occasionally, I see in the error logs that indicate that the counters have failed to open because they had already been used in the current process. I presume this is due to my .NET appdomain having been reset within the same w3wp.exe process. How can I avoid these errors and re-establish a connection to my performance counters when my app domain has been recycled?

Counter construction:

PerformanceCounter pc = new PerformanceCounter();
pc.CategoryName = category_name;
pc.CounterName = counter_name;
pc.ReadOnly = false;
pc.InstanceLifetime =
PerformanceCounterInstanceLifetime.Process;
pc.InstanceName = instance_name;

Counter usage:

pc.Increment()

[Update on 3/26/09] The error message received is:

Instance '_lm_w3svc_1_root_myapp' already exists with a lifetime of Process. It cannot be recreated or reused until it has been removed or until the process using it has exited. already exists with a lifetime of Process.

I tried to replicate the exception in a console application by initializing the performance counters and writing to one of them in a transient AppDomain. I then unload the AppDomain and do it again in a second Appdomain (same process). They both succeed. I'm unsure now exactly what the cause of this is, my assumption about AppDomain recycling in ASP.NET seems to be false.

Answer 1

The above error may also be experienced in the Application Logs when using process based WCF performance counters. The symptoms of this are a block of three errors in the Application event logs:

A performance counter was not loaded.
Category Name: ServiceModelService 3.0.0.0
Counter Name: Calls
Exception:System.InvalidOperationException: Instance 'abc@def|service.svc' already exists with a lifetime of Process. It cannot be recreated or reused until it has been removed or until the process using it has exited.

This always occurs immediately after the following Information message in the System event logs:

A worker process with process id of 'nnnn' serving application pool 'MyAppPool' has requested a recycle because the worker process reached its allowed processing time limit.

This causes the performance monitor counters for this service to become unavailable, apparently for a few hours.

The ServiceModelService 3.0.0.0 version number will depend on the version of .NET you are using (this was tested using .NET 3.5).

Background

The fault is triggered by the worker process reaching its processing time limit, at which point it must be recycled. This is set in the IIS Application Pool Recycling settings (IIS 6.0 and above, therefore Windows Server 2003 and above). Recycling of the worker process causes the new process based performance counter name to conflict with the old, which creates an error. This is because IIS uses overlapped recycling, where the worker process to be terminated is kept running until after the new worker process is started.

Reproduction

(Tested on Windows Server 2003 = IIS 6.0)

• Create a WCF service.
• Add the following to the web.config in the <system.serviceModel> section:
  <diagnostics performanceCounters="All" />
• In the Application Pool properties for the service under Properties → Recycling, set the Recycle worker process to 1 minute. Manually recycling the Application Pool has no effect, as this does not create a request from the worker process to recycle (as evident in the Event Viewer System logs with W3SVC Information events).
• In soapUI create a Test Suite / Test Case / Test Step / Test Request for the WCF operation.
• Add the test case to either a load test in soapUI, or use loadUI, firing the call at the rate of 1 per second.
• Every 1 minute the worker process will request a recycle (evident in the System logs) . Every 2 minutes this will result in a batch of three errors in the Application logs from System.ServiceModel 3.0.0.0.
• The performance counters for that service will become unavailable until the worker process recycles again. (NB Setting the recycle period to a higher value at this point to see how long the performance counters are unavailable for will actually recycle the processes and make the counters available again.)

Possible solutions

Solution 1 - the red herrings

Hotfix KB981574 supersedes hotfix KB971601. The latter hotfix describes the problem:

FIX: The performance counters that monitor an application stop responding when the application exits and restarts and you receive the System.InvalidOperationException exception on a computer that is running .NET Framework 2.0

Applying the former hotfix does not remedy the problem. Applying the latter hotfix caused app pool errors.

Solution 2 - a working solution

It is possible to create a service host factory that exposes a custom service host:

using System;
using System.Diagnostics;
using System.ServiceModel;
using System.ServiceModel.Activation;

namespace MyNamespace
{
    public class WebFarmServiceHostFactory : ServiceHostFactory
    {
        protected override ServiceHost CreateServiceHost(
                   Type serviceType, Uri[] baseAddresses)
        {
            return new WebFarmServiceHost(serviceType, baseAddresses);
        }
    }

    public class WebFarmServiceHost : ServiceHost
    {
        public WebFarmServiceHost(
            Type serviceType, params Uri[] baseAddresses)
            : base(serviceType, baseAddresses) { }

        protected override void ApplyConfiguration()
        {
            base.ApplyConfiguration();

            Description.Name = "W3wp" + Process.GetCurrentProcess().Id +
                               Description.Name;
        }
    }
}

And refer to the factory in the service markup .svc file:

<%@ ServiceHost Language="C#" Debug="true" Factory="MyNamespace.WebFarmServiceHostFactory" Service="WcfService1.Service1" CodeBehind="Service1.svc.cs" %>

Final tweaks

Unfortunately, although this makes the problem occur much less frequently, it still occurs (as an approximation around 30 times less, although I've not measured accurate stats on this).

A simple tweak which seems to remedy the problem completely is to add a sleep command just before base.ApplyConfiguration();:

Thread.Sleep(1);

This only fires once per worker process recycle, so should have negligible impact on the performance of the service. You may have to raise this value (you could make it configurable) although the minimum setting of 1ms worked for me (debate of how long this command actually sleeps for aside).

Solution 3 - the simplest fix

In IIS there is a DisallowOverlappingRotation Metabase Property. This can be set as follows (example given for MyAppPool application pool):

cscript %SYSTEMDRIVE%\inetpub\adminscripts\adsutil.vbs SET w3svc/AppPools/MyAppPool/DisallowOverlappingRotation TRUE

Comparison of solutions

Solution #3 will cause your site to be down longer when a worker process recycles due to the absence of IIS overlapped recycling.

On soapUI load tests of a basic web service using 100+ transactions per second with 5 threads, a 'freeze' where new transactions were blocked was evident for a couple of seconds every time the worker process recycled. This freeze was more prolonged (8+ seconds) when a more complex web service was tested.

Solution #2 produced no such blocking, had a smooth flow of responses during the recycle, and produced no perfmon conflict errors.

Conclusion

For web services where low latency is not a requirement, Solution #3 could be used. You could even set the recycling to be daily at a set time if you know the load distribution and quiet times of day (this can be done in the same tab in IIS). This could even be staggered if a web farm was used.

For web services which cannot tolerate such delays, it seems Solution #2 is the best way forwards.