Missed Garbage Collection Notifications

Question

We are running a web farm using .NET. Each web server holds a considerable amount of static objects in it's memory. A Gen 2 garbage collection (GC) takes 10-20 seconds and it runs every 5 minutes. We ran more or less into the same problems that StackOverflow ran into: http://samsaffron.com/archive/2011/10/28/in-managed-code-we-trust-our-recent-battles-with-the-net-garbage-collector

At the moment, we are reducing the number of objects in the cache. However, this takes time.

At the same time, we implemented the methods documented here to get notifications in .NET about approaching GCs. The goal is to take a web server out of the farm when a GC is approaching and include it into the farm after the GC is over. However, we only get a notification for 0.7% of all GCs. We are using a maxGenerationThreshold and largeObjectHeapThreshold of 8. We tried other thresholds, but the amount of missed GCs did not change.

We are using concurrent server garbage collection (http://msdn.microsoft.com/en-us/library/ms229357.aspx). The GCLatencyMode is Interactive (see http://msdn.microsoft.com/en-us/library/system.runtime.gclatencymode.aspx). Here again, we tried to use other GC modes (Workstation mode, Batch, etc.). And again we did not get a notification for most of the GCs.

Are we doing something wrong, or is it impossible to get a notification for every GC that occurs? How can we increase the number of notifications?

According to http://assets.red-gate.com/community/books/assets/Under_the_Hood_of_.NET_Management.pdf, at the beginning a GC is triggered when Gen2 hits ~10 MB. We have a lot of RAM, so if we could set this threshold manually to a higher level, it would take more time to reach this threshold and in my understanding the probability would increase to get a notification. Is there a way to modify this threshold?

This is the code that registers and listens to the notifications:

GC.RegisterForFullGCNotification(gcThreshold, gcThreshold);
// Start a thread using WaitForFullGCProc.
thWaitForFullGC = new Thread(WaitForFullGCProc);
thWaitForFullGC.Name = "HealthTestGCNotificationListenerThread (Threshold=" + gcThreshold + ")";
thWaitForFullGC.IsBackground = true;

WaitForFullGCProc():

    private void WaitForFullGCProc()
{
    try
    {
        while (!gcAbort)
        {
            // Check for a notification of an approaching collection.
            GCNotificationStatus s;
            do
            {
                int timeOut = CheckForMissedGc() > 0 ? 5000 : (10 * 60 * 1000);
                s = GC.WaitForFullGCApproach(timeOut);
                if (this.GcState == GCState.InducedUnnotified)
                {
                    // Set the GcState back to okay to prevent the message from staying in the ApplicationMonitoring.
                    this.GcState = GCState.Okay;
                }
            } while (s == GCNotificationStatus.Timeout);

            if (s == GCNotificationStatus.Succeeded)
            {
                SetGcState(GCState.Approaching, "GC is approaching..");
                gcApproachNotificationCount++;
            }
            else
            {
                ...
            }

            Stopwatch stopwatch = Stopwatch.StartNew();
            s = GC.WaitForFullGCComplete((int)PrewarnTime.TotalMilliseconds);
            long elapsed = stopwatch.ElapsedMilliseconds;

            if (s == GCNotificationStatus.Timeout)
            {
                if (this.ForceGCWhenApproaching && !this.IsInGc && !this.IsPeriodicGcApproaching)
                {
                    this.IsInGc = true;
                    GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced, blocking: true);
                    GC.WaitForPendingFinalizers();
                    elapsed = stopwatch.ElapsedMilliseconds;
                    this.IsInGc = false;
                }
            }
        }
        gcAbort = false;
    }
    catch (Exception e)
    {
    }
}

Answer 1

Note: This is more of a comment but includes a large code sample.

Have you considered trying to get your GC notifications through another way? Jeffrey Richter ( CLR via C#) explains a good way to get notifications,it uses an object and checks its finalizer method in what generation it is.

This is the class: It uses internal objects which are either collected if the supplied generation matches ( See new GenObject(0); for example. ) or resurrected for the next higher generation.

And you just subscribe to it with GCNotification.GCDone += GCDoneHandler;

 public static class GCNotification
    {
        private static Action<Int32> s_gcDone = null; // The event's field
        public static event Action<Int32> GCDone
        {
            add
            {
                // If there were no registered delegates before, start reporting notifications now
                if (s_gcDone == null) { new GenObject(0); new GenObject(1); new GenObject(2); }
                s_gcDone += value;
            }
            remove { s_gcDone -= value; }
        }
        private sealed class GenObject
        {
            private Int32 m_generation;
            public GenObject(Int32 generation) { m_generation = generation; }
            ~GenObject()
            { // This is the Finalize method
                // If this object is in the generation we want (or higher),
                // notify the delegates that a GC just completed
                if (GC.GetGeneration(this) >= m_generation)
                {
                    Action<Int32> temp = Volatile.Read(ref s_gcDone);
                    if (temp != null) temp(m_generation);
                }
                // Keep reporting notifications if there is at least one delegate registered,
                // the AppDomain isn't unloading, and the process isn’t shutting down
                if ((s_gcDone != null)
                && !AppDomain.CurrentDomain.IsFinalizingForUnload()
                && !Environment.HasShutdownStarted)
                {
                    // For Gen 0, create a new object; for Gen 2, resurrect the object
                    // & let the GC call Finalize again the next time Gen 2 is GC'd
                    if (m_generation == 0) new GenObject(0);
                    else GC.ReRegisterForFinalize(this);
                }
                else { /* Let the objects go away */ }
            }
        }
    }