Microsoft.Web.Administration memory leak

Question

Have a windows service that, amongst other things, monitors IIS application pools. If any pool is has configured applications and is not running, it (the pool) is started. This has been working fine for some time. It was recently discovered that the service was leaking memory. Looking at a memory dump the culprit is the Microsoft.Web.Administration used to check the application pools. The only object that's disposable is ServerManager and I have that in a using block. Have found other reports of this leak but no solution yet. (see user comments in http://msdn.microsoft.com/en-us/library/microsoft.web.administration.servermanager(v=vs.90).aspx#CommunityContent)

When dump all the roots for Microsoft.Web.Administration.ServerManager (481 in this dump) I only see roots one of them. Assume this is the current iteration.

Not sure why these Web.Administration objects cannot be collected, even though they don't seem to be rooted(?). Any ideas on how to plug this leak?

I moved the code to a console application and reproduced the leak there.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using Microsoft.Web.Administration;

namespace Web.Administration.LeakExample
{
    public static class ServerManagerExtensions
    {
        /// <summary>
        /// Returns the application counts for application pools under the specified ServerManager.
        /// Application pools without applications will not be returned.
        /// </summary>
        /// <param name="manager"></param>
        /// <returns></returns>
        public static Dictionary<string, uint> GetPoolApplicationCounts(this ServerManager manager)
        {
            if (manager == null)
            {
                throw new ArgumentNullException();
            }

            var appCounts = new Dictionary<string, uint>(manager.ApplicationPools.Count);
            foreach (var app in manager.Sites.SelectMany(site => site.Applications))
            {
                if (!appCounts.ContainsKey(app.ApplicationPoolName))
                {
                    appCounts.Add(app.ApplicationPoolName, 1);
                }
                else
                {
                    appCounts[app.ApplicationPoolName]++;
                }
            }

            return appCounts;
        }
    }

    class Program
    {
        static void Main(string[] args)
        {
            while (!Console.KeyAvailable)
            {
                Console.WriteLine("Checking App Pools...");

                using (var manager = new ServerManager())
                {
                    var appCounts = manager.GetPoolApplicationCounts();

                    var appPools = manager.ApplicationPools.Where(pool => appCounts.ContainsKey(pool.Name));

                    foreach (
                        var pool in
                            appPools.Where(
                                pool => (pool.State != ObjectState.Started) && (pool.State != ObjectState.Starting)))
                    {
                        var state = pool.Start();

                        if ((state == ObjectState.Started) || (state == ObjectState.Starting))
                        {
                            Console.WriteLine("Started app pool \"{0}\"", pool.Name);
                        }
                        else
                        {
                            Console.WriteLine("Failed to start app pool \"{0}\". state:{1}", pool.Name, state);
                        }
                    }
                }

                Console.WriteLine("Sleeping...");
                Thread.Sleep(1000);
            }
        }
    }
}

WinDbg session:

> !dumpheap -stat

606b778c      479         5748 System.Runtime.Remoting.Activation.ConstructionLevelActivator
00472088      480         5760 System.Collections.Generic.Dictionary`2+ValueCollection[[System.String, mscorlib],[Microsoft.Web.Administration.Interop.IAppHostElement, Microsoft.Web.Administration]]
00471d5c      480         5760 System.Collections.Generic.SortedList`2+ValueList[[System.String, mscorlib],[Microsoft.Web.Administration.Configuration, Microsoft.Web.Administration]]
00194568      481         5772 Web.Administration.LeakExample.Program+<>c__DisplayClass3
00198438      480         7680 Microsoft.Web.Administration.Interop.AppHostWritableAdminManager
0047199c      481         7696 System.Linq.Enumerable+<>c__DisplayClassf`1[[Microsoft.Web.Administration.ApplicationPool, Microsoft.Web.Administration]]
606cc200      958        11496 System.Char
00471e2c      479        11496 System.Collections.Generic.SortedList`2+SortedListValueEnumerator[[System.String, mscorlib],[Microsoft.Web.Administration.Configuration, Microsoft.Web.Administration]]
0047090c      479        11496 System.Collections.Generic.List`1+Enumerator[[Microsoft.Web.Administration.Site, Microsoft.Web.Administration]]
004706f0      958        11496 Microsoft.Web.Administration.ConfigurationElementCollectionBase`1+<>c__DisplayClass4[[Microsoft.Web.Administration.Site, Microsoft.Web.Administration]]
00471aec      480        11520 System.Collections.Generic.List`1+Enumerator[[Microsoft.Web.Administration.ApplicationPool, Microsoft.Web.Administration]]
0047041c      480        11520 System.Collections.Generic.List`1[[Microsoft.Web.Administration.Site, Microsoft.Web.Administration]]
00196d58      480        11520 System.Collections.Generic.List`1[[Microsoft.Web.Administration.Configuration, Microsoft.Web.Administration]]
004715cc      481        11544 System.Collections.Generic.List`1[[Microsoft.Web.Administration.ApplicationPool, Microsoft.Web.Administration]]
0047103c     1437        17244 Microsoft.Web.Administration.ConfigurationElementCollectionBase`1+<>c__DisplayClass4[[Microsoft.Web.Administration.Application, Microsoft.Web.Administration]]
606ccfc8     1438        17256 System.Int32
00196a58      480        19200 System.Collections.Generic.SortedList`2[[System.String, mscorlib],[Microsoft.Web.Administration.Configuration, Microsoft.Web.Administration]]
0019660c      480        21120 Microsoft.Web.Administration.ConfigurationManager
00471130      958        22992 System.Collections.Generic.List`1+Enumerator[[Microsoft.Web.Administration.Application, Microsoft.Web.Administration]]
00470d68      960        23040 System.Collections.Generic.List`1[[Microsoft.Web.Administration.Application, Microsoft.Web.Administration]]
00198558      480        23040 System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[Microsoft.Web.Administration.Interop.IAppHostElement, Microsoft.Web.Administration]]
00196a08      481        23088 Microsoft.Web.Administration.Configuration
00183170      481        23088 System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[System.UInt32, mscorlib]]
606b7574      479        24908 System.Runtime.Remoting.Messaging.ConstructorCallMessage
00199e7c      479        24908 System.Linq.Enumerable+<SelectManyIterator>d__14`2[[Microsoft.Web.Administration.Site, Microsoft.Web.Administration],[Microsoft.Web.Administration.Application, Microsoft.Web.Administration]]
00199384      480        28800 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[Microsoft.Web.Administration.Interop.IAppHostElement, Microsoft.Web.Administration]][]
00470874      958        30656 System.Predicate`1[[Microsoft.Web.Administration.Site, Microsoft.Web.Administration]]
004718a8      962        30784 System.Linq.Enumerable+WhereEnumerableIterator`1[[Microsoft.Web.Administration.ApplicationPool, Microsoft.Web.Administration]]
001944d8      481        30784 Microsoft.Web.Administration.ServerManager
00195178      963        30816 System.Func`2[[Microsoft.Web.Administration.ApplicationPool, Microsoft.Web.Administration],[System.Boolean, mscorlib]]
606ccf90      968        31268 System.Int32[]
00197a50      480        32640 Microsoft.Web.Administration.SiteCollection
00194da0      481        32708 Microsoft.Web.Administration.ApplicationPoolCollection
004719f8     2874        34488 Microsoft.Web.Administration.ConfigurationElementCollectionBase`1+<>c__DisplayClass4[[Microsoft.Web.Administration.ApplicationPool, Microsoft.Web.Administration]]
0019810c      960        42240 Microsoft.Web.Administration.ConfigurationSection
00471098     1437        45984 System.Predicate`1[[Microsoft.Web.Administration.Application, Microsoft.Web.Administration]]
606cc1c8      961        46638 System.Char[]
001838d0      481        59644 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[System.UInt32, mscorlib]][]
606bedd0     5269        63228 System.UInt32
00470a7c      960        69120 Microsoft.Web.Administration.ApplicationCollection
00197c60      960        76800 Microsoft.Web.Administration.Site
00197ea8     1440        86400 Microsoft.Web.Administration.Application
00471a54     2874        91968 System.Predicate`1[[Microsoft.Web.Administration.ApplicationPool, Microsoft.Web.Administration]]
6067c684     4832       141272 System.Object[]
00195024     2886       196248 Microsoft.Web.Administration.ApplicationPool
606c7b20    16323       261168 System.__ComObject

>.foreach (obj {!dumpheap -mt 001944d8 -short}){!gcroot -all obj}

    002ef124 003f0359 Web.Administration.LeakExample.Program.Main(System.String[]) [c:\Users\Chuck\Documents\Visual Studio 2012\Projects\Web.Administration.LeakExample\Web.Administration.LeakExample\Program.cs @ 74]
        ebp+70: 002ef148
            ->  02773270 System.Linq.Enumerable+WhereEnumerableIterator`1[[Microsoft.Web.Administration.ApplicationPool, Microsoft.Web.Administration]]
            ->  02772948 Microsoft.Web.Administration.ApplicationPoolCollection
            ->  02772060 Microsoft.Web.Administration.ServerManager

    002ef124 003f0359 Web.Administration.LeakExample.Program.Main(System.String[]) [c:\Users\Chuck\Documents\Visual Studio 2012\Projects\Web.Administration.LeakExample\Web.Administration.LeakExample\Program.cs @ 74]
        ebp+68: 002ef150
            ->  02773270 System.Linq.Enumerable+WhereEnumerableIterator`1[[Microsoft.Web.Administration.ApplicationPool, Microsoft.Web.Administration]]
            ->  02772948 Microsoft.Web.Administration.ApplicationPoolCollection
            ->  02772060 Microsoft.Web.Administration.ServerManager

    002ef124 003f0359 Web.Administration.LeakExample.Program.Main(System.String[]) [c:\Users\Chuck\Documents\Visual Studio 2012\Projects\Web.Administration.LeakExample\Web.Administration.LeakExample\Program.cs @ 74]
        ebp+78: 002ef140
            ->  027732c0 System.Linq.Enumerable+WhereEnumerableIterator`1[[Microsoft.Web.Administration.ApplicationPool, Microsoft.Web.Administration]]
            ->  02772948 Microsoft.Web.Administration.ApplicationPoolCollection
            ->  02772060 Microsoft.Web.Administration.ServerManager

    002ef124 003f0359 Web.Administration.LeakExample.Program.Main(System.String[]) [c:\Users\Chuck\Documents\Visual Studio 2012\Projects\Web.Administration.LeakExample\Web.Administration.LeakExample\Program.cs @ 74]
        ebp+6c: 002ef14c
            ->  02773270 System.Linq.Enumerable+WhereEnumerableIterator`1[[Microsoft.Web.Administration.ApplicationPool, Microsoft.Web.Administration]]
            ->  02772948 Microsoft.Web.Administration.ApplicationPoolCollection
            ->  02772060 Microsoft.Web.Administration.ServerManager

    002ef124 003f0359 Web.Administration.LeakExample.Program.Main(System.String[]) [c:\Users\Chuck\Documents\Visual Studio 2012\Projects\Web.Administration.LeakExample\Web.Administration.LeakExample\Program.cs @ 74]
        ebp+74: 002ef144
            ->  027732c0 System.Linq.Enumerable+WhereEnumerableIterator`1[[Microsoft.Web.Administration.ApplicationPool, Microsoft.Web.Administration]]
            ->  02772948 Microsoft.Web.Administration.ApplicationPoolCollection
            ->  02772060 Microsoft.Web.Administration.ServerManager

    002ef124 003f0359 Web.Administration.LeakExample.Program.Main(System.String[]) [c:\Users\Chuck\Documents\Visual Studio 2012\Projects\Web.Administration.LeakExample\Web.Administration.LeakExample\Program.cs @ 74]
        ebp+60: 002ef158
            ->  02772948 Microsoft.Web.Administration.ApplicationPoolCollection
            ->  02772060 Microsoft.Web.Administration.ServerManager

    002ef124 003f0359 Web.Administration.LeakExample.Program.Main(System.String[]) [c:\Users\Chuck\Documents\Visual Studio 2012\Projects\Web.Administration.LeakExample\Web.Administration.LeakExample\Program.cs @ 74]
        ebp+3c: 002ef17c
            ->  02772060 Microsoft.Web.Administration.ServerManager

    002ef124 003f0359 Web.Administration.LeakExample.Program.Main(System.String[]) [c:\Users\Chuck\Documents\Visual Studio 2012\Projects\Web.Administration.LeakExample\Web.Administration.LeakExample\Program.cs @ 74]
        ebp+4c: 002ef16c
            ->  027732c0 System.Linq.Enumerable+WhereEnumerableIterator`1[[Microsoft.Web.Administration.ApplicationPool, Microsoft.Web.Administration]]
            ->  02772948 Microsoft.Web.Administration.ApplicationPoolCollection
            ->  02772060 Microsoft.Web.Administration.ServerManager

    002ef124 003f0359 Web.Administration.LeakExample.Program.Main(System.String[]) [c:\Users\Chuck\Documents\Visual Studio 2012\Projects\Web.Administration.LeakExample\Web.Administration.LeakExample\Program.cs @ 74]
        ebp+40: 002ef178
            ->  02773270 System.Linq.Enumerable+WhereEnumerableIterator`1[[Microsoft.Web.Administration.ApplicationPool, Microsoft.Web.Administration]]
            ->  02772948 Microsoft.Web.Administration.ApplicationPoolCollection
            ->  02772060 Microsoft.Web.Administration.ServerManager

    002ef124 003f0359 Web.Administration.LeakExample.Program.Main(System.String[]) [c:\Users\Chuck\Documents\Visual Studio 2012\Projects\Web.Administration.LeakExample\Web.Administration.LeakExample\Program.cs @ 74]
        ebp+50: 002ef168
            ->  02772060 Microsoft.Web.Administration.ServerManager

Found 10 roots.

Answer 1

When I first read the question, the first thing that occurred to me was that this whole issue could possibly be avoided entirely by using the Windows Activation Service (WAS).

My first pass through the code that you posted gave me the feeling that your issue could be related to accessing/modifying of the various closures that are being created by the foreach loops.

Try these small refactorings and see if any/all are able to reduce or eliminate the problem or failing that, provides more info on the root cause:

1. avoid deferred execution of your LINQ queries by calling .ToList() immediately
2. Instead of re-using the ServerManager object, create a new one on each iteration of the loop

3. Since you're only using ApplicationPool.Name, refactor this:

   foreach (var app in manager.Sites.SelectMany(site => site.Applications))

to this:

foreach (var app in manager.Sites.SelectMany(site => site.Applications, 
   (s, a) => a.Name)
    .ToList())

My hypothesis is that this is what is happening:

• New instance of ServerManager is created in the using statement
• Static extension method is called on that object reference, which uses ApplicationPools internally
• ApplicationPools is immediately used again in the construction of an IQueryable, which is immediately used (but not fully executed -- foreach statements use the yield return internally) and filtered.
• Call to .Start() changes (mutates) state of the ApplicationPool referenced by the ServerManager object reference
• the rest isn't relevant, so the loop goes back to the beginning. It still uses the same ServerManager. Since that reference has mutated state (because of call to .Start()), it may be getting flagged by the GC and never collected - that app pool is never returned in the results of subsequent iterations, and may cause orphaning of the reference in memory.

This is just a hypothesis, keep in mind!