Memory leak while using Threads

Question

I appear to have a memory leak in this piece of code. It is a console app, which creates a couple of classes (WorkerThread), each of which writes to the console at specified intervals. The Threading.Timer is used to do this, hence writing to the console is performed in a separate thread (the TimerCallback is called in a seperate thread taken from the ThreadPool). To complicate matters, the MainThread class hooks on to the Changed event of the FileSystemWatcher; when the test.xml file changes, the WorkerThread classes are recreated.

Each time the file is saved, (each time that the WorkerThread and therefore the Timer is recreated), the memory in the Task Manager increases (Mem Usage, and sometimes also VM Size); furthermore, in .Net Memory Profiler (v3.1), the Undisposed Instances of the WorkerThread class increases by two (this may be a red herring though, because I've read that .Net Memory Profiler had a bug whereby it struggled to detect disposed classes.

Anyway, here's the code - does anyone know what's wrong?

EDIT: I've moved the class creation out of the FileSystemWatcher.Changed event handler, meaning that the WorkerThread classes are always being created in the same thread. I've added some protection to the static variables. I've also provided threading information to show more clearly what's going on, and have been interchanging using the Timer with using an explicit Thread; however, the memory is still leaking! The Mem Usage increases slowly all the time (is this simply due to extra text in the console window?), and the VM Size increases when I change the file. Here is the latest version of the code:

EDIT This appears to be primarily a problem with the console using up memory, as you write to it. There is still a problem with explicitly written Threads increasing the memory usage. See my answer below.

class Program
{
    private static List<WorkerThread> threads = new List<WorkerThread>();

    static void Main(string[] args)
    {
        MainThread.Start();

    }
}

public class MainThread
{
    private static int _eventsRaised = 0;
    private static int _eventsRespondedTo = 0;
    private static bool _reload = false;
    private static readonly object _reloadLock = new object();
    //to do something once in handler, though
    //this code would go in onStart in a windows service.
    public static void Start()
    {
        WorkerThread thread1 = null;
        WorkerThread thread2 = null;

        Console.WriteLine("Start: thread " + Thread.CurrentThread.ManagedThreadId);
        //watch config
        FileSystemWatcher watcher = new FileSystemWatcher();
        watcher.Path = "../../";
        watcher.Filter = "test.xml";
        watcher.EnableRaisingEvents = true;
        //subscribe to changed event. note that this event can be raised a number of times for each save of the file.
        watcher.Changed += (sender, args) => FileChanged(sender, args);

        thread1 = new WorkerThread("foo", 10);
        thread2 = new WorkerThread("bar", 15);

        while (true)
        {
            if (_reload)
            {
                //create our two threads.
                Console.WriteLine("Start - reload: thread " + Thread.CurrentThread.ManagedThreadId);
                //wait, to enable other file changed events to pass
                Console.WriteLine("Start - waiting: thread " + Thread.CurrentThread.ManagedThreadId);
                thread1.Dispose();
                thread2.Dispose();
                Thread.Sleep(3000); //each thread lasts 0.5 seconds, so 3 seconds should be plenty to wait for the 
                                    //LoadData function to complete.
                Monitor.Enter(_reloadLock);
                thread1 = new WorkerThread("foo", 10);
                thread2 = new WorkerThread("bar", 15);
                _reload = false;
                Monitor.Exit(_reloadLock);
            }
        }
    }

    //this event handler is called in a separate thread to Start()
    static void FileChanged(object source, FileSystemEventArgs e)
    {
        Monitor.Enter(_reloadLock);
        _eventsRaised += 1;
        //if it was more than a second since the last event (ie, it's a new save), then wait for 3 seconds (to avoid 
        //multiple events for the same file save) before processing
        if (!_reload)
        {
            Console.WriteLine("FileChanged: thread " + Thread.CurrentThread.ManagedThreadId);
            _eventsRespondedTo += 1;
            Console.WriteLine("FileChanged. Handled event {0} of {1}.", _eventsRespondedTo, _eventsRaised);
            //tell main thread to restart threads
            _reload = true;
        }
        Monitor.Exit(_reloadLock);
    }
}

public class WorkerThread : IDisposable
{
    private System.Threading.Timer timer;   //the timer exists in its own separate thread pool thread.
    private string _name = string.Empty;
    private int _interval = 0;  //thread wait interval in ms.
    private Thread _thread = null;
    private ThreadStart _job = null;

    public WorkerThread(string name, int interval)
    {
        Console.WriteLine("WorkerThread: thread " + Thread.CurrentThread.ManagedThreadId);
        _name = name;
        _interval = interval * 1000;
        _job = new ThreadStart(LoadData);
        _thread = new Thread(_job);
        _thread.Start();
        //timer = new Timer(Tick, null, 1000, interval * 1000);
    }

    //this delegate instance does NOT run in the same thread as the thread that created the timer. It runs in its own
    //thread, taken from the ThreadPool. Hence, no need to create a new thread for the LoadData method.
    private void Tick(object state)
    {
        //LoadData();
    }

    //Loads the data. Called from separate thread. Lasts 0.5 seconds.
    //
    //private void LoadData(object state)
    private void LoadData()
    {
        while (true)
        {
            for (int i = 0; i < 10; i++)
            {
                Console.WriteLine(string.Format("Worker thread {0} ({2}): {1}", _name, i, Thread.CurrentThread.ManagedThreadId));
                Thread.Sleep(50);
            }
            Thread.Sleep(_interval);
        }
    }

    public void Stop()
    {
        Console.WriteLine("Stop: thread " + Thread.CurrentThread.ManagedThreadId);
        //timer.Dispose();
        _thread.Abort();
    }


    #region IDisposable Members

    public void Dispose()
    {
        Console.WriteLine("Dispose: thread " + Thread.CurrentThread.ManagedThreadId);
        //timer.Dispose();
        _thread.Abort();
    }

    #endregion
}

Answer 1

You have two issues, both separate:

In Watcher.Changed's handler you call Thread.Sleep(3000); This is poor behaviour in a callback of a thread you do not own (since it is being supplied by the pool owned/used by the watcher. This is not the source of your problem though. This it in direct violation of the guidelines for use

You use statics all over the place which is horrible, and has likely led you into this problem:

static void test()
{
    _eventsRaised += 1;
    //if it was more than a second since the last event (ie, it's a new save), then wait for 3 seconds (to avoid 
    //multiple events for the same file save) before processing
    if (DateTime.Now.Ticks - _lastEventTicks > 1000)
    {
        Thread.Sleep(3000);
        _lastEventTicks = DateTime.Now.Ticks;
        _eventsRespondedTo += 1;
        Console.WriteLine("File changed. Handled event {0} of {1}.", _eventsRespondedTo, _eventsRaised);
        //stop threads and then restart them
        thread1.Stop();
        thread2.Stop();
        thread1 = new WorkerThread("foo", 20);
        thread2 = new WorkerThread("bar", 30);
    }
}

This callback can fire repeatedly on multiple different threads (it uses the system thread pool for this) You code assumes that only one thread will ever execute this method at a time since threads can be created but not not stopped.

Imagine: thread A and B

1. A thread1.Stop()
2. A thread2.Stop()
3. B thread1.Stop()
4. B thread2.Stop()
5. A thread1 = new WorkerThread()
6. A thread2 = new WorkerThread()
7. B thread1 = new WorkerThread()
8. B thread2 = new WorkerThread()

You now have 4 WorkerThread instances on the heap but only two variables referencing them, the two created by A have leaked. The event handling and callback registration with the timer means that theses leaked WorkerThreads are kept alive (in the GC sense) despite you having no reference to them in your code. they stay leaked for ever.

There are other flaws in the design but this is a critical one.