FileSystemWatcher skips some events

Question

If you google for FileSystemWatcher issues, you will find a lot of articles about FileSystemWatcher skipping some events (not firing all events). Basically, if you change a lot of files in watched folder some of them will not be processes by FileSystemWatcher.

Why is that so, and how can I avoid missing events?

Answer 1

Cause

FileSystemWatcher is watching for changes happening in some folder. When file is changed (e.g. file is created), the FileSystemWatcher raises the appropriate event. The event handler might unzip the file, read its content to decide how to process it further, write record of it in database log table and move the file to another folder. The processing of the file might take some time.

During that time another file might be created in watched folder. Since FileSystemWatcher’s event handler is processing the first file, it cannot handle creation event of second file. So, the second file is missed by FileSystemWatcher.

Solution

Since file processing might take some time and creation of other files might get undetected by FileSystemWatcher, file processing should be separated from file change detection and file change detection should be so short that it never misses single file change. File handling can be divided into two threads: one for the file change detection and the other for the file processing. When file is changed and it is detected by FileSystemWatcher, appropriate event handler should only read its path, forward it to file processing thread and close itself so FileSystemWatcher can detect another file change and use the same event handler. The processing thread could take as much time as it needs to process the file. A queue is used for forwarding file path from event handler thread to the processing thread.

This is classic producer-consumer problem. More about producer-consumer queue can be found here.

Code

using System;
using System.IO;
using System.Threading;
using System.Collections.Generic;

namespace FileSystemWatcherExample {
    class Program {
        static void Main(string[] args) {
            // If a directory and filter are not specified, exit program
            if (args.Length !=2) {
                // Display the proper way to call the program
                Console.WriteLine("Usage: Watcher.exe \"directory\" \"filter\"");
                return;
            }

            FileProcessor fileProcessor = new FileProcessor();

            // Create a new FileSystemWatcher
            FileSystemWatcher fileSystemWatcher1 = new FileSystemWatcher();

            // Set FileSystemWatcher's properties
            fileSystemWatcher1.Path = args[0];
            fileSystemWatcher1.Filter = args[1];
            fileSystemWatcher1.IncludeSubdirectories = false;

            // Add event handlers
            fileSystemWatcher1.Created += new System.IO.FileSystemEventHandler(this.fileSystemWatcher1_Created);

            // Start to watch
            fileSystemWatcher1.EnableRaisingEvents = true;

            // Wait for the user to quit the program
            Console.WriteLine("Press \'q\' to quit the program.");
            while(Console.Read()!='q');

            // Turn off FileSystemWatcher
            if (fileSystemWatcher1 != null) {
                fileSystemWatcher1.EnableRaisingEvents = false;
                fileSystemWatcher1.Dispose();
                fileSystemWatcher1 = null;
            }

            // Dispose fileProcessor
            if (fileProcessor != null)
                fileProcessor.Dispose();
        }

        // Define the event handler
        private void fileSystemWatcher1_Created(object sender, FileSystemEventArgs e) {
            // If file is created...
            if (e.ChangeType == WatcherChangeTypes.Created) {
                // ...enqueue it's file name so it can be processed...
                fileProcessor.EnqueueFileName(e.FullPath);
            }
            // ...and immediately finish event handler
        }
    }


    // File processor class
    class FileProcessor : IDisposable {
        // Create an AutoResetEvent EventWaitHandle
        private EventWaitHandle eventWaitHandle = new AutoResetEvent(false);
        private Thread worker;
        private readonly object locker = new object();
        private Queue<string> fileNamesQueue = new Queue<string>();

        public FileProcessor() {
            // Create worker thread
            worker = new Thread(Work);
            // Start worker thread
            worker.Start();
        }

        public void EnqueueFileName(string FileName) {
            // Enqueue the file name
            // This statement is secured by lock to prevent other thread to mess with queue while enqueuing file name
            lock (locker) fileNamesQueue.Enqueue(FileName);
            // Signal worker that file name is enqueued and that it can be processed
            eventWaitHandle.Set();
        }

        private void Work() {
            while (true) {
                string fileName = null;

                // Dequeue the file name
                lock (locker)
                    if (fileNamesQueue.Count > 0) {
                        fileName = fileNamesQueue.Dequeue();
                        // If file name is null then stop worker thread
                        if (fileName == null) return;
                    }

                if (fileName != null) {
                    // Process file
                    ProcessFile(fileName);
                } else {
                    // No more file names - wait for a signal
                    eventWaitHandle.WaitOne();
                }
            }
        }

        private ProcessFile(string FileName) {
            // Maybe it has to wait for file to stop being used by process that created it before it can continue
            // Unzip file
            // Read its content
            // Log file data to database
            // Move file to archive folder
        }


        #region IDisposable Members

        public void Dispose() {
            // Signal the FileProcessor to exit
            EnqueueFileName(null);
            // Wait for the FileProcessor's thread to finish
            worker.Join();
            // Release any OS resources
            eventWaitHandle.Close();
        }

        #endregion
    }
}