Why does IHttpAsyncHandler leak memory under load?

Question

I have noticed that the .NET IHttpAsyncHandler (and the IHttpHandler, to a lesser degree) leak memory when subjected to concurrent web requests.

In my tests, the Visual Studio web server (Cassini) jumps from 6MB memory to over 100MB, and once the test is finished, none of it is reclaimed.

The problem can be reproduced easily. Create a new solution (LeakyHandler) with two projects:

1. An ASP.NET web application (LeakyHandler.WebApp)
2. A Console application (LeakyHandler.ConsoleApp)

In LeakyHandler.WebApp:

1. Create a class called TestHandler that implements IHttpAsyncHandler.
2. In the request processing, do a brief Sleep and end the response.
3. Add the HTTP handler to Web.config as test.ashx.

In LeakyHandler.ConsoleApp:

1. Generate a large number of HttpWebRequests to test.ashx and execute them asynchronously.

As the number of HttpWebRequests (sampleSize) is increased, the memory leak is made more and more apparent.

LeakyHandler.WebApp > TestHandler.cs

namespace LeakyHandler.WebApp
{
    public class TestHandler : IHttpAsyncHandler
    {
        #region IHttpAsyncHandler Members

        private ProcessRequestDelegate Delegate { get; set; }
        public delegate void ProcessRequestDelegate(HttpContext context);

        public IAsyncResult BeginProcessRequest(HttpContext context, AsyncCallback cb, object extraData)
        {
            Delegate = ProcessRequest;
            return Delegate.BeginInvoke(context, cb, extraData);
        }

        public void EndProcessRequest(IAsyncResult result)
        {
            Delegate.EndInvoke(result);
        }

        #endregion

        #region IHttpHandler Members

        public bool IsReusable
        {
            get { return true; }
        }

        public void ProcessRequest(HttpContext context)
        {
            Thread.Sleep(10);
            context.Response.End();
        }

        #endregion
    }
}

LeakyHandler.WebApp > Web.config

<?xml version="1.0"?>

<configuration>
    <system.web>
        <compilation debug="false" />
        <httpHandlers>
            <add verb="POST" path="test.ashx" type="LeakyHandler.WebApp.TestHandler" />
        </httpHandlers>
    </system.web>
</configuration>

LeakyHandler.ConsoleApp > Program.cs

namespace LeakyHandler.ConsoleApp
{
    class Program
    {
        private static int sampleSize = 10000;
        private static int startedCount = 0;
        private static int completedCount = 0;

        static void Main(string[] args)
        {
            Console.WriteLine("Press any key to start.");
            Console.ReadKey();

            string url = "http://localhost:3000/test.ashx";
            for (int i = 0; i < sampleSize; i++)
            {
                HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
                request.Method = "POST";
                request.BeginGetResponse(GetResponseCallback, request);

                Console.WriteLine("S: " + Interlocked.Increment(ref startedCount));
            }

            Console.ReadKey();
        }

        static void GetResponseCallback(IAsyncResult result)
        {
            HttpWebRequest request = (HttpWebRequest)result.AsyncState;
            HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(result);
            try
            {
                using (Stream stream = response.GetResponseStream())
                {
                    using (StreamReader streamReader = new StreamReader(stream))
                    {
                        streamReader.ReadToEnd();
                        System.Console.WriteLine("C: " + Interlocked.Increment(ref completedCount));
                    }
                }
                response.Close();
            }
            catch (Exception ex)
            {
                System.Console.WriteLine("Error processing response: " + ex.Message);
            }
        }
    }
}

Debugging Update

I used WinDbg to look into the dump files, and a few suspicious types are being held in memory and never released. Each time I run a test with a sample size of 10,000, I end up with 10,000 more of these objects being held in memory.

• System.Runtime.Remoting.ServerIdentity
• System.Runtime.Remoting.ObjRef
• Microsoft.VisualStudio.WebHost.Connection
• System.Runtime.Remoting.Messaging.StackBuilderSink
• System.Runtime.Remoting.ChannelInfo
• System.Runtime.Remoting.Messaging.ServerObjectTerminatorSink

These objects lie in the Generation 2 heap and are not collected, even after a forced full garbage collection.

Important Note

The problem exists even when forcing sequential requests and even without the Thread.Sleep(10) in ProcessRequest, it's just a lot more subtle. The example exacerbates the problem by making it more readily apparent, but the fundamentals are the same.

Answer 1

I've had a look at your code (and run it) and I don't believe the increasing memory you are seeing is actually a memory leak.

The problem you've got is that your calling code (the console app) is essentially running in a tight loop.

However, your handler has to process each request, and is additionally being "nobbled" by the Thread.Sleep(10). The practical upshot of this is that your handler can't keep up with the requests coming in, so its "working set" grows and grows as more requests queue up, waiting to be processed.

I took your code and added an AutoResetEvent to the console app, doing a

.WaitOne() after request.BeginGetResponse(GetResponseCallback, request);

and a

.Set() after streamReader.ReadToEnd();

This has the effect of synchronising the calls so the next call can't be made until the first call has called back (and completed). The behaviour you are seeing goes away.

In summary, I think this is purely an runaway situation and not actually a memory leak at all.

Note: I monitored the memory with the following, in the GetResponseCallback method:

 GC.Collect();
 GC.WaitForPendingFinalizers();
 Console.WriteLine(GC.GetTotalMemory(true));

[Edit in response to comment from Anton] I'm not suggesting there is no problem here at all. If your usage scenario is such that this hammering of the handler is a real usage scenario, then clearly you have an issue. My point is that it is not a memeory leak issue, but a capacity issue. The way to approach solving this would be, maybe, to write a handler that could run faster, or to scale out to multiple servers, etc, etc.

A leak is when resources are held onto after they are finished with, increasing the size of the working set. These resources have not been "finished with", they are in a queue and waiting to be serviced. Once they are complete I believe they are being released correctly.

[Edit in response to Anton's further comments] OK - I've uncovered something! I think this is a Cassini issue that doesn't occur under IIS. Are you running your handler under Cassini (The Visual Studio Development Web Server)?

I too see these leaky System.Runtime.Remoting namespace instances when I am running under Cassini only. I do not see them if I set the handler up to run under IIS. Can you confirm if this is the case for you?

This reminds me of some other remoting/Cassini issue I've seen. IIRC, having an instance of something like an IPrincipal that needs to exist in the BeginRequest of a module, and also at the end of the module lifecycle, needs to derive from MarshalByRefObject in Cassini but not IIS. For some reason it seems Cassini is doing some remoting internally that IIS isn't.