How to diagnose source of Handle leak

Question

The Problem

I just put in some performance logging yesterday as I noticed a handle leak from watching Task Manager quite some time ago, though fixing it has been low priority. This is an overnight run with a sample every 10 seconds.

I haven't run this to failure yet, due to time constraints and my test computer is also my dev computer so running this while writing code is not ideal... so I'm not sure if/when it will crash, but I highly suspect it's only a matter of time.

Note: The red boxed in region is where I "stopped" the working loop and restarted it after a short pause. Threads dropped on the "stop" from ~100 down to ~20. The Handles did not drop until the loop was restarted after about 30 seconds from ~62,000 to ~40,000. So some handles are getting GC'd, just not nearly as many as I expect should be. I can't figure out what root is preventing all these handles from getting collected or where they are originally coming from (ie. Tasks, GUI, Files etc.).

If you already have an idea what could be causing this issue, then no need to read any further. I have provided the the rest of this info and code for reference in a shot-gun style approach of sussing out the issue. I will remove, edit, etc. as the root cause is narrowed down. By the same token, if something of interest is missing let me know and I will try to provide it (logs, dumps, etc.).

---

What I've Done

On my own I've gone through this tutorial on Tracking Handle Misuse and gotten as far as looking at the dump files to find where the Handles Open and Close... however it was just too overwhelming with thousands of handles to make any sense of and I had trouble getting Symbols to load so the pointers were just gibberish to me.

I have yet to go through the following two on my list, but wondered if there were some friendlier methods first...

• Debug Leaky Apps: Identify And Prevent Memory Leaks In Managed Code
• Tracking down managed memory leaks (how to find a GC leak)

I've also split out the code I suspected to be the potential causes of this into another small application and everything appeared to get Garbage Collected without issue (albeit the execution pattern was greatly simplified compared to the real app).

Potential Culprits

I do have several long-lived instanced classes that last as long as the application is open for, including 5 Forms that are created only once each and then hidden/shown as needed. I use a main object as my application controller and then Models and Views are wired up via events to Presenters in a Presenter-First pattern.

Below are some things I do in this application, which may or may not be important:

• Use custom Action, Func and lambdas extensively, some of which may be long-lived
• 3 custom delegates for events and which can spawn Tasks for async execution.
• Extension for safely invoking on Controls.
• Very, very heavily use Task and Parallel.For/Parallel.Foreach to run worker methods (or events as mentioned above)
• Never use Thread.Sleep(), but instead a custom Sleep.For() which uses an AutoResetEvent.

Main Loop

The general flow of this application when it is Running is based on a loop over a series of files in the Offline version and the polling of a digital input signal in the Online version. Below is the sudo-code with comments for the Offline version which is what I can run from my laptop without the need for external hardware and what the chart above was monitoring (I don't have access to the hardware for Online mode at this time).

public void foo()
{
    // Sudo Code
    var InfiniteReplay = true;
    var Stopped = new CancellationToken();
    var FileList = new List<string>();
    var AutoMode = new ManualResetEvent(false);
    var CompleteSignal = new ManualResetEvent(false);
    Action<CancellationToken> PauseIfRequired = (tkn) => { };

    // Enumerate a Directory...

    // ... Load each file and do work
    do
    {
        foreach (var File in FileList)
        {
            /// Method stops the loop waiting on a local AutoResetEvent
            /// if the CompleteSignal returns faster than the
            /// desired working rate of ~2 seconds
            PauseIfRequired(Stopped);

            /// While not 'Stopped', poll for Automatic Mode
            /// NOTE: This mimics how the online system polls a digital
            /// input instead of a ManualResetEvent.
            while (!Stopped.IsCancellationRequested)
            {
                if (AutoMode.WaitOne(100))
                {
                    /// Class level Field as the Interface did not allow
                    /// for passing the string with the event below
                    m_nextFile = File;

                    // Raises Event async using Task.Factory.StartNew() extension
                    m_acquireData.Raise();
                    break;
                }
            }

            // Escape if Canceled
            if (Stopped.IsCancellationRequested)
                break;

            // If In Automatic Mode, Wait for Complete Signal
            if (AutoMode.WaitOne(0))
            {
                // Ensure Signal Transition
                CompleteSignal.WaitOne(0);
                if (!CompleteSignal.WaitOne(10000))
                {
                    // Log timeout and warn User after 10 seconds, then continue looping
                }
            }
        }
        // Keep looping through same set of files until 'Stopped' if in Infinite Replay Mode
    } while (!Stopped.IsCancellationRequested && InfiniteReplay);
}

Async Events

Below is the extension for events and most are executed using the default asynchronous option. The 'TryRaising()' extensions just wrap the delegates in a try-catch and logs any exceptions (while they do not re-throw it isn't part of normal program flow for them to be responsible for catching exceptions).

using System.Threading.Tasks;
using System;

namespace Common.EventDelegates
{
    public delegate void TriggerEvent();
    public delegate void ValueEvent<T>(T p_value) where T : struct;
    public delegate void ReferenceEvent<T>(T p_reference);

    public static partial class DelegateExtensions
    {
        public static void Raise(this TriggerEvent p_response, bool p_synchronized = false)
        {
            if (p_response == null)
                return;

            if (!p_synchronized)
                Task.Factory.StartNew(() => { p_response.TryRaising(); });
            else
                p_response.TryRaising();
        }

        public static void Broadcast<T>(this ValueEvent<T> p_response, T p_value, bool p_synchronized = false)
            where T : struct
        {
            if (p_response == null)
                return;

            if (!p_synchronized)
                Task.Factory.StartNew(() => { p_response.TryBroadcasting(p_value); });
            else
                p_response.TryBroadcasting(p_value);
        }

        public static void Send<T>(this ReferenceEvent<T> p_response, T p_reference, bool p_synchronized = false)
            where T : class
        {
            if (p_response == null)
                return;

            if (!p_synchronized)
                Task.Factory.StartNew(() => { p_response.TrySending(p_reference); });
            else
                p_response.TrySending(p_reference);
        }
    }
}

GUI Safe-Invoke

using System;
using System.Windows.Forms;
using Common.FluentValidation;
using Common.Environment;

namespace Common.Extensions
{
    public static class InvokeExtensions
    {
        /// <summary>
        /// Execute a method on the control's owning thread.
        /// </summary>
        /// http://stackoverflow.com/q/714666
        public static void SafeInvoke(this Control p_control, Action p_action, bool p_forceSynchronous = false)
        {
            p_control
                .CannotBeNull("p_control");

            if (p_control.InvokeRequired)
            {
                if (p_forceSynchronous)
                    p_control.Invoke((Action)delegate { SafeInvoke(p_control, p_action, p_forceSynchronous); });
                else
                    p_control.BeginInvoke((Action)delegate { SafeInvoke(p_control, p_action, p_forceSynchronous); });
            }
            else
            {
                if (!p_control.IsHandleCreated)
                {
                    // The user is responsible for ensuring that the control has a valid handle
                    throw
                        new
                            InvalidOperationException("SafeInvoke on \"" + p_control.Name + "\" failed because the control had no handle.");

                    /// jwdebug
                    /// Only manually create handles when knowingly on the GUI thread
                    /// Add the line below to generate a handle http://stackoverflow.com/a/3289692/1718702
                    //var h = this.Handle;
                }

                if (p_control.IsDisposed)
                    throw
                        new
                            ObjectDisposedException("Control is already disposed.");

                p_action.Invoke();
            }
        }
    }
}

Sleep.For()

using System.Threading;
using Common.FluentValidation;

namespace Common.Environment
{
    public static partial class Sleep
    {
        public static bool For(int p_milliseconds, CancellationToken p_cancelToken = default(CancellationToken))
        {
            // Used as "No-Op" during debug
            if (p_milliseconds == 0)
                return false;

            // Validate
            p_milliseconds
                .MustBeEqualOrAbove(0, "p_milliseconds");

            // Exit immediate if cancelled
            if (p_cancelToken != default(CancellationToken))
                if (p_cancelToken.IsCancellationRequested)
                    return true;

            var SleepTimer =
                new AutoResetEvent(false);

            // Cancellation Callback Action
            if (p_cancelToken != default(CancellationToken))
                p_cancelToken
                    .Register(() => SleepTimer.Set());

            // Block on SleepTimer
            var Canceled = SleepTimer.WaitOne(p_milliseconds);

            return Canceled;
        }
    }
}

Answer 1

All the comments so far have been quite helpful and I have found at least one source of my handle leaks to be the Sleep.For() method. I still think I have handles leaking, but at a significantly slower rate and I also understand better now why they were leaking.

It had to do with the scope of the passed in token and cleaning up the local token inside the method in a using statement. Once I fixed this, I started seeing all those unnamed Event handles in Process Explorer being created and destroyed instead of just sitting there.

As an aside, I found Anatomy of a "Memory Leak" late last night and will definitely be learning more about Windbg for further investigations.

I am also doing a long-running performance test again to see if this was the only leak or not and reviewing other sections of my code that use WaitHandles to make sure I properly scope and dispose of them.

Fixed Sleep.For()

using System.Threading;
using Common.FluentValidation;
using System;

namespace Common.Environment
{
    public static partial class Sleep
    {
        /// <summary>
        /// Block the current thread for a specified amount of time.
        /// </summary>
        /// <param name="p_milliseconds">Time to block for.</param>
        /// <param name="p_cancelToken">External token for waking thread early.</param>
        /// <returns>True if sleeping was cancelled before timer expired.</returns>
        public static bool For(int p_milliseconds, CancellationToken p_cancelToken = default(CancellationToken))
        {
            // Used as "No-Op" during debug
            if (p_milliseconds == 0)
                return false;

            // Validate
            p_milliseconds
                .MustBeEqualOrAbove(0, "p_milliseconds");

            // Merge Tokens and block on either
            CancellationToken LocalToken = new CancellationToken();
            using (var SleeperSource = CancellationTokenSource.CreateLinkedTokenSource(LocalToken, p_cancelToken))
            {
                SleeperSource
                    .Token
                    .WaitHandle
                    .WaitOne(p_milliseconds);

                return SleeperSource.IsCancellationRequested;
            }
        }
    }
}

Test App (Console)

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Common.Environment;
using System.Threading;

namespace HandleTesting
{
    class Program
    {
        private static CancellationTokenSource static_cts = new CancellationTokenSource();

        static void Main(string[] args)
        {
            //Periodic.StartNew(() =>
            //{
            //    Console.WriteLine(string.Format("CPU_{0} Mem_{1} T_{2} H_{3} GDI_{4} USR_{5}",
            //        Performance.CPU_Percent_Load(),
            //        Performance.PrivateMemorySize64(),
            //        Performance.ThreadCount(),
            //        Performance.HandleCount(),
            //        Performance.GDI_Objects_Count(),
            //        Performance.USER_Objects_Count()));
            //}, 5);

            Action RunMethod;
            Console.WriteLine("Program Started...\r\n");
            var MainScope_cts = new CancellationTokenSource();
            do
            {
                GC.Collect();
                GC.WaitForPendingFinalizers();
                GC.Collect();

                try
                {
                    var LoopScope_cts = new CancellationTokenSource();
                    Console.WriteLine("Enter number of Sleep.For() iterations:");
                    var Loops = int.Parse(Console.ReadLine());

                    Console.WriteLine("Enter millisecond interval per iteration:");
                    var Rate = int.Parse(Console.ReadLine());

                    RunMethod = () => SomeMethod(Loops, Rate, MainScope_cts.Token);

                    RunMethod();
                }
                catch (Exception ex)
                {
                    Console.WriteLine(ex.Message);
                }
                Console.WriteLine("\r\nPress any key to try again, or press Escape to exit.");
            }
            while (Console.ReadKey().Key != ConsoleKey.Escape);
            Console.WriteLine("\r\nProgram Ended...");
        }

        private static void SomeMethod(int p_loops, int p_rate, CancellationToken p_token)
        {
            var local_cts = new CancellationTokenSource();
            Console.WriteLine("Method Executing " + p_loops + " Loops at " + p_rate + "ms each.\r\n");
            for (int i = 0; i < p_loops; i++)
            {
                var Handles = Performance.HandleCount();
                Sleep.For(p_rate, p_token); /*<--- Change token here to test GC and variable Scoping*/
                Console.WriteLine("H_pre " + Handles + ", H_post " + Performance.HandleCount());
            }
        }
    }
}

Performance (Helper Class)

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Runtime.InteropServices;
using System.Management;
using Common.Extensions;
using System.Diagnostics;

namespace Common.Environment
{
    public static partial class Performance
    {
        //https://stackoverflow.com/a/9543180/1718702
        [DllImport("User32")]
        extern public static int GetGuiResources(IntPtr hProcess, int uiFlags);

        public static int GDI_Objects_Count()
        {
            //Return the count of GDI objects.
            return GetGuiResources(System.Diagnostics.Process.GetCurrentProcess().Handle, 0);
        }
        public static int USER_Objects_Count()
        {
            //Return the count of USER objects.
            return GetGuiResources(System.Diagnostics.Process.GetCurrentProcess().Handle, 1);
        }
        public static string CPU_Percent_Load()
        {
            //http://allen-conway-dotnet.blogspot.ca/2013/07/get-cpu-usage-across-all-cores-in-c.html
            //Get CPU usage values using a WMI query
            ManagementObjectSearcher searcher = new ManagementObjectSearcher("SELECT * FROM Win32_PerfFormattedData_PerfOS_Processor");
            var cpuTimes = searcher.Get()
                .Cast<ManagementObject>()
                .Select(mo =>
                    new
                    {
                        Name = mo["Name"],
                        Usage = mo["PercentProcessorTime"]
                    }
                ).ToList();

            var Total = cpuTimes[cpuTimes.Count - 1];
            cpuTimes.RemoveAt(cpuTimes.Count - 1);

            var PercentUsage = string.Join("_", cpuTimes.Select(x => Convert.ToInt32(x.Usage).ToString("00")));

            return PercentUsage + "," + Convert.ToInt32(Total.Usage).ToString("00");
        }
        public static long PrivateMemorySize64()
        {
            using (var P = Process.GetCurrentProcess())
            {
                return P.PrivateMemorySize64;
            }
        }
        public static int ThreadCount()
        {
            using (var P = Process.GetCurrentProcess())
            {
                return P.Threads.Count;
            }
        }
        public static int HandleCount()
        {
            using (var P = Process.GetCurrentProcess())
            {
                return P.HandleCount;
            }
        }
    }
}

Update 2013-10-18:

Results of the long run. No other code changes were required to fix this.