Is it really that expensive to increase FileSystemWatcher.InternalBufferSize?

Question

I'm using a FileSystemWatcher to monitor changes in a folder, but as soon as I have more than a few hundred modifications in a short time, I miss some of them because the internal buffer overflows. So I want to increase the InternalBufferSize (I know it won't really solve the problem, but it will make it less likely to occur), but I see this warning in the documentation:

However, increasing buffer size is expensive, because it comes from non-paged memory that cannot be swapped out to disk, so keep the buffer as small as possible.

So my question is: does it really matter? Most computers today have at least 1GB of RAM, so it seems to me that if I set the buffer size to 1MB (instead of the default 8KB), it shouldn't really matter if 1MB can't be swapped out to disk. Or am I missing something? I don't know much about low level stuff such as paged/non-paged memory, so I'm not sure what the impact would be...

Answer 1

The memory in which the buffer gets allocated is certainly a precious resource. Windows will not deal with exhausting the memory pool well, drivers will start to fail at random. The size of the pool is dynamically set (but can be changed) and depends on the amount of available RAM.

The default buffer size that FSW asks for is 8192 bytes. Not much on modern machines. The underlying winapi function will not allow you to ask for more than 64KB. An entry is the buffer is 12 bytes plus the length of the file path times two. So worse case is 8192 / (12 + 260*2) = 15 notifications before the buffer runs out. That should work in most circumstances without much trouble, unless you monitor an entire drive or have very high disk traffic in the directory you are watching. In which case asking for a bigger buffer is fair. There is no golden formula, be sure to implement the FileSystemWatcher.Error event so you know that you've got a buffer problem.

In most practical cases you need to deal with the FSW events carefully. They will be raised while a process still has a lock on a file. So doing things like opening or copying the file are troublesome. You deal with that by putting the notifications on a thread-safe queue and use another thread to try to acquire a lock on the file, repeatedly if necessary. Such a queue is now automatically also a very good way to quickly empty the buffer. The only thing you've got to watch for now is that the queue doesn't blow up beyond reasonable proportions that will make your program crash with OOM.

Answer 2

Consider a consumer producer design to read in the FileSystemWatcher events.

BlockingCollection Overview

If you have some FileSystemWatcher events you don't need to process then quickly dismiss them.
Or if some can process faster than others have a separate collection to keep the total count down.

Answer 3

Non-paged memory has limited size (update: modern versions of Windows don't have as strict limit as in previous versions, and the amount of memory is now a flexible value that depends on the overall RAM available to Windows) and is very important to kernel-mode inhabitants (device drivers, OS itself). Consuming it without severe control can quickly cause instability in the system and, what is worse, you won't find what has caused this instability.

Also, as comments suggest, it's not the buffer size that matters, but how fast you remove data from the buffer.

In most cases, a filesystem filter driver will do a better job than FileSystemWatcher. The benefits are that you can have any log buffer that you need (as you would create it in whatever memory you need and not be limited by non-paged memory) and also you can handle events as they happen, not after they happen. Also, you can filter requests with finer granularity than FileSystemWatcher lets you.