Using FileSystemWatcher with multiple files

Question

I want to use FileSystemWatcher to monitor a directory and its subdirectories for files that are moved. And then I want to trigger some code when all the files have been moved. But I don't know how. My code as is will trigger each time a file is moved, and if a user moves several files at once I only want it to trigger once for all files. So basically I want to create a list, and once the moving of all files is done I want to do stuff to that list...

Here's the code:

class Monitor
{
    private List<string> _filePaths;  
    public void CreateWatcher(string path)
    {
        FileSystemWatcher watcher = new FileSystemWatcher();

        watcher.Filter = "*.*";

        watcher.Created += new
        FileSystemEventHandler(watcher_FileCreated);

        watcher.Path = path;
        watcher.IncludeSubdirectories = true;

        watcher.EnableRaisingEvents = true;
    }

    void watcher_FileCreated(object sender, FileSystemEventArgs e)
    {
        _filePaths.Add(e.FullPath);
        Console.WriteLine("Files have been created or moved!");
    }

}

UPDATE: Trying to use Chris's code, but it doesn't work (see my comment at Chris's answer):

class Monitor
    {
        private List<string> _filePaths;
        private Timer _notificationTimer;
        private FileSystemWatcher _fsw;
        public Monitor(string path)
        {
            _notificationTimer = new Timer();
            _notificationTimer.Elapsed += notificationTimer_Elapsed;
            // CooldownSeconds is the number of seconds the Timer is 'extended' each time a file is added.
            // I found it convenient to put this value in an app config file.
            int CooldownSeconds = 1;
            _notificationTimer.Interval = CooldownSeconds * 1000;

            _fsw = new FileSystemWatcher();
            _fsw.Path = path;
            _fsw.IncludeSubdirectories = true;
            _fsw.EnableRaisingEvents = true;

            // Set up the particulars of your FileSystemWatcher.
            _fsw.Created += fsw_Created;
        }

        private void notificationTimer_Elapsed(object sender, ElapsedEventArgs e)
        {
            //
            // Do what you want to do with your List of files.
            //
            Console.Write("Done");
            // Stop the timer and wait for the next batch of files.            
            _notificationTimer.Stop();
            // Clear your file List.
            _filePaths = new List<string>();
        }


        // Fires when a file is created.
        private void fsw_Created(object sender, FileSystemEventArgs e)
        {
            // Add to our List of files.
            _filePaths.Add(e.Name);

            // 'Reset' timer.
            _notificationTimer.Stop();
            _notificationTimer.Start();
        }


    }

UPDATE 2:

Tried this according to Anders's answer:

public class FileListEventArgs : EventArgs
{
    public List<string> FileList { get; set; }
}

public class Monitor
{
    private List<string> filePaths;
    private ReaderWriterLockSlim rwlock;
    private Timer processTimer;
    public event EventHandler FileListCreated;


    public void OnFileListCreated(FileListEventArgs e)
    {
        if (FileListCreated != null)
            FileListCreated(this, e);
    }

    public Monitor(string path)
    {
        filePaths = new List<string>();

        rwlock = new ReaderWriterLockSlim();

        FileSystemWatcher watcher = new FileSystemWatcher();
        watcher.Filter = "*.*";
        watcher.Created += watcher_FileCreated;

        watcher.Path = path;
        watcher.IncludeSubdirectories = true;
        watcher.EnableRaisingEvents = true;
    }

    private void ProcessQueue()
    {
        List<string> list = new List<string>();
        try
        {
            Console.WriteLine("Processing queue, " + filePaths.Count + " files created:");
            rwlock.EnterReadLock();

        }
        finally
        {
            if (processTimer != null)
            {
                processTimer.Stop();
                processTimer.Dispose();
                processTimer = null;
                OnFileListCreated(new FileListEventArgs { FileList = filePaths });
                filePaths.Clear();
            }
            rwlock.ExitReadLock();
        }
    }

    void watcher_FileCreated(object sender, FileSystemEventArgs e)
    {
        try
        {
            rwlock.EnterWriteLock();
            filePaths.Add(e.FullPath);

            if (processTimer == null)
            {
                // First file, start timer.
                processTimer = new Timer(2000);
                processTimer.Elapsed += (o, ee) => ProcessQueue();
                processTimer.Start();
            }
            else
            {
                // Subsequent file, reset timer. 
                processTimer.Stop();
                processTimer.Start();
            }

        }
        finally
        {
            rwlock.ExitWriteLock();
        }
    }

I had to move the event trigger into the finally statement, and that works. I don't know if there is some reason I wouldn't want to do that?

Answer 1

I had to do the exact same thing. Use a System.Timers.Timer in your Monitor class and code its Elapsed event to process your List of files and clear the List. When the first item is added to your file List via the FSW events, start the Timer. When subsequent items are added to the list 'reset' the Timer by stopping and restarting it.

Something like this:

class Monitor
{
    FileSystemWatcher _fsw;
    Timer _notificationTimer;
    List<string> _filePaths = new List<string>();

    public Monitor() {
        _notificationTimer = new Timer();
        _notificationTimer.Elapsed += notificationTimer_Elapsed;
        // CooldownSeconds is the number of seconds the Timer is 'extended' each time a file is added.
        // I found it convenient to put this value in an app config file.
        _notificationTimer.Interval = CooldownSeconds * 1000;

        _fsw = new FileSystemWatcher();
        // Set up the particulars of your FileSystemWatcher.
        _fsw.Created += fsw_Created;
    }

    private void notificationTimer_Elapsed(object sender, ElapsedEventArgs e) {
        //
        // Do what you want to do with your List of files.
        //

        // Stop the timer and wait for the next batch of files.            
        _notificationTimer.Stop();
        // Clear your file List.
        _filePaths = new List<string>();
    }


    // Fires when a file is created.
    private void fsw_Created(object sender, FileSystemEventArgs e) {
        // Add to our List of files.
        _filePaths.Add(e.Name);

        // 'Reset' timer.
        _notificationTimer.Stop();
        _notificationTimer.Start();
    }
}

Answer 2

Also, set buffer size to larger than default to avoid buffer overflow. It happens when more than 25 files are dropped in source directory (in my test). If 200 files dropped, event handler is only called for few files, not all.

_watcher.InternalBufferSize = 65536; //Max size of buffer

Answer 3

Like Jay says: a timer is probably the only way to "group" events. The lock may be overkill but I don't like the idea of mutating a collection in a multithreaded situation (I think the events from the fswatcher are called on threads from a pool).

  public class Monitor : IDisposable
  {
     private List<string> filePaths;
     private ReaderWriterLockSlim rwlock;
     private Timer processTimer;
     private string watchedPath;
     private FileSystemWatcher watcher;

     public Monitor(string watchedPath)
     {
        filePaths = new List<string>();

        rwlock = new ReaderWriterLockSlim();

        this.watchedPath = watchedPath;
        InitFileSystemWatcher();
     }

     private void InitFileSystemWatcher()
     {
        watcher = new FileSystemWatcher();
        watcher.Filter = "*.*";
        watcher.Created += Watcher_FileCreated;
        watcher.Error += Watcher_Error;
        watcher.Path = watchedPath;
        watcher.IncludeSubdirectories = true;
        watcher.EnableRaisingEvents = true;
     }

     private void Watcher_Error(object sender, ErrorEventArgs e)
     {
        // Watcher crashed. Re-init.
        InitFileSystemWatcher();
     }

     private void Watcher_FileCreated(object sender, FileSystemEventArgs e)
     {
        try
        {
           rwlock.EnterWriteLock();
           filePaths.Add(e.FullPath);

           if (processTimer == null)
           {
              // First file, start timer.
              processTimer = new Timer(2000);
              processTimer.Elapsed += ProcessQueue;
              processTimer.Start();
           }
           else
           {
              // Subsequent file, reset timer.
              processTimer.Stop();
              processTimer.Start();
           }

        }
        finally
        {
           rwlock.ExitWriteLock();
        }
     }

     private void ProcessQueue(object sender, ElapsedEventArgs args)
     {
        try
        {
           Console.WriteLine("Processing queue, " + filePaths.Count + " files created:");
           rwlock.EnterReadLock();
           foreach (string filePath in filePaths)
           {
              Console.WriteLine(filePath);
           }
           filePaths.Clear();
        }
        finally
        {
           if (processTimer != null)
           {
              processTimer.Stop();
              processTimer.Dispose();
              processTimer = null;
           }
           rwlock.ExitReadLock();
        }
     }

     protected virtual void Dispose(bool disposing)
     {
        if (disposing)
        {
           if (rwlock != null)
           {
              rwlock.Dispose();
              rwlock = null;
           }
           if (watcher != null)
           {
              watcher.EnableRaisingEvents = false;
              watcher.Dispose();
              watcher = null;
           }
        }
     }

     public void Dispose()
     {
        Dispose(true);
        GC.SuppressFinalize(this);
     }

  }     

Remember to set the buffer size on your fswatcher AND implement "resurrection" of the fswatcher if it gets an error (i.e. bind the error event to a method that recreates the watcher).

Edit: note, the timer in this example is a System.Timers.Timer, not a System.Threading.Timer

Edit: Now contains error handling for the watcher, dispose logic.

Answer 4

Rx - throttle - makes this job easy. Here is a testable, reusable solution.

public class Files
{
     public static FileSystemWatcher WatchForChanges(string path, string filter, Action triggeredAction)
            {
                var monitor = new FileSystemWatcher(path, filter);

                //monitor.NotifyFilter = NotifyFilters.FileName;
                monitor.Changed += (o, e) => triggeredAction.Invoke();
                monitor.Created += (o, e) => triggeredAction.Invoke();
                monitor.Renamed += (o, e) => triggeredAction.Invoke();
                monitor.EnableRaisingEvents = true;

                return monitor;
            }
}

Lets merge events into a single sequence

  public IObservable<Unit> OnUpdate(string path, string pattern)
        {
            return Observable.Create<Unit>(o =>
            {
                var watcher = Files.WatchForChanges(path, pattern, () => o.OnNext(new Unit()));

                return watcher;
            });
        }

then finally, the usage

OnUpdate(path, "*.*").Throttle(Timespan.FromSeconds(10)).Subscribe(this, _ => DoWork())

u can obviously play with the throttle speed to suit your needs.