FileSystemWatcher for notification

Question

I'm creating a Windows application which use FileSystemWatcher. FileSystemWatcher watches some catalog for changes. And every time some file is added to this catalog FileSystemWatcher must add information about this file to an XML file. All works fine, but, when I add, for example, 100 files at same time (say that some application adds these files to the catalog) not every file's information appears in this xml file.

I want to use Queue. And use it to add items to this collection. And use a timer. The timer will add information to the XML from this collection. Is it good idea?

Can anyone advise me what to do?

---

So i think that i must create a Windows application and a Windows Service. WinApp will only add information to EventLog and Windows Service will read information EventLog and write it to XML. I think it will the best way to do it. I'm waiting for good advice

Answer 1

As Michael Stum writes in his answer, you could try increasing the buffer size (FileSystemWatcher.InternalBufferSize). Note however that you shouldn't set this value to too high a value. Also, IMHO this is possibly only a temporary fix, see what happens when you add yet more files to your folder at the same time.

I have read about some other things you can attempt if increasing the buffer size doesn't help:

• If you subscribe to the notification events of FileSystemWatcher, try to keep your event handler as short as possible; ie. make sure execution doesn't stay there long. If you have to do a fair amount of work per file notification, you could try to start a separate thread and do the processing there; your event handler could then get back to the caller very quickly (and the file notification would get removed from the buffer/queue sooner).

• Don't use any additional information provided by FileSystemWatcher apart from the basic notification that something has changed. Once you get any file change notification, wait a brief timespan until no more notifications arrive (ie. wait for the last notification out of 100 simultaneous file change notifications). Then list the contents of the directory manually and transfer the required information to your XML.

  This has one major drawback: Manually detecting whether, and which, files have been deleted, renamed, or created is not easy. Your program would have to keep the last directory listing against which the current listing can be compared to find out what's changed.

  The advantage of this is that you can be fairly sure that no changes will be dropped due to some FileSystemWatcher buffer overflow.

Answer 2

FileSystemWatcher has an internal Buffer for changes. When there are a lot of quick changes, the buffer may not catch all Events.

Try increasing the InternalBufferSize to something higher.

You can set the buffer to 4 KB or larger, but it must not exceed 64 KB. For best performance, use a multiple of 4 KB on Intel-based computers.

The system notifies the component of file changes, and it stores those changes in a buffer the component creates and passes to the APIs. Each event can use up to 16 bytes of memory, not including the file name. If there are many changes in a short time, the buffer can overflow. This causes the component to lose track of changes in the directory, and it will only provide blanket notification. Increasing the size of the buffer has the following ramifications:

Increasing the buffer size can prevent missing file system change events. Note that an instance of the FileSystemWatcher class can raise an Error event when an event is missed or when the buffer size is exceeded, due to dependencies with the Windows operating system.

Increasing buffer size is expensive, as it comes from non paged memory that cannot be swapped out to disk, so keep the buffer as small as possible. To avoid a buffer overflow, use the NotifyFilter and IncludeSubdirectories properties to filter out unwanted change notifications.

For diagnostics, maybe you should first subscribe to the Error Event to see if it's really a Buffer Overflow.

Also as said, set the NotifyFilter to the smallest required flags, which could be NotifyFilters.LastWrite if you only want to track changes.