FileSystemWatcher not working properly

Question

I am using FileSystemWatcher to scan a folder where files are being uploaded from the web application. I have more than 1000 files being uploaded to the folder. Now, the problem is that the FileSystemWatcher started when the first file is copied, sometimes, it doesn't work for all the files. Any suggestions will be highly appreciable.

Answer 1

The FileSystemWatcher documentation says that (emphasis mine):

The Windows operating system notifies your component of file changes in a buffer created by the FileSystemWatcher. If there are many changes in a short time, the buffer can overflow. This causes the component to lose track of changes in the directory, and it will only provide blanket notification. Increasing the size of the buffer with the InternalBufferSize property is expensive, as it comes from non-paged memory that cannot be swapped out to disk, so keep the buffer as small yet large enough to not miss any file change events. To avoid a buffer overflow, use the NotifyFilter and IncludeSubdirectories properties so you can filter out unwanted change notifications.

.

Note that a FileSystemWatcher may miss an event when the buffer size is exceeded. To avoid missing events, follow these guidelines:

Increase the buffer size by setting the InternalBufferSize property.

Avoid watching files with long file names, because a long file name contributes to filling up the buffer. Consider renaming these files using shorter names.

Keep your event handling code as short as possible.

If you have tried the above and it's still not reliable enough for you, I 'm afraid the only solution will be to subscribe to the Error event and manually enumerate the directory contents whenever it occurs.

Answer 2

To elaborate on Jon's answer:

There is another possible solution to postpone a buffer overflow from occuring. By handling the events you monitor, the Created event for example, as quickly as possibly you remove the information tied to the event from the buffer so that it can reclaim this space.

To speed this up you can opt to asynchronously process the files. As soon as the event is triggered you queue the full path to the file in a Queue which in turn will be processed by a worker thread. The duty of the event handler tied to the Created event is reduced to adding a string (the file path) to a queue.

For example:

public class FileProcessor
{
    private readonly Queue<string> files = new Queue<string>();

    public void EnqueueFile(string path)
    {
        files.Enqueue(path);
    }
}

The code for the event handler attached to the created event can then be minimized to this:

static void file_Created(object sender, FileSystemEventArgs e)
{
    _fileProcessor.EnqueueFile(e.FullPath);
}

You can find more information about this (including sample code) in a post I wrote about two years ago:

https://github.com/geersch/FileSystemWatcher