After FileSystemWatcher fires - Thread Pool or Dedicated thread?

Question

I am about to implement the archetypal FileSystemWatcher solution. I have a directory to monitor for file creations, and the task of sucking up created files and inserting the into a DB. Roughly this will involve reading and processing 6 or 7, 80 char text files that appear at a rate of 150mS in bursts that occur every couple of seconds, and rarely a 2MB binary file will also have to be processed. This will most likely be a 24/7 process.

From what I have read about the FileSystemWatcher object it is better to enqueue its events in one thread and then dequeue/process them in another thread. The quandary I have right now is what would be the better creation mechanism of the thread that does the processing. The choices I can see are:

1. Each time I get a FSW event I manually create a new thread (yeah I know .. stupid architecture, but I had to say it).

2. Throw the processing at the CLR thread pool whenever I get an FSW event

3. On start up, create a dedicated second thread for the processing and use a producer/consumer model to handle the work. The main thread enqueues the request and the second thread dequeues it and performs the work.

I am tending towards the third method as the preferred one as I know the work thread will always be required - and also probably more so because I have no feel for the thread pool.

Answer 1

If you know that the second thread will always be required, and you also know that you'll never need more than one worker thread, then option three is good enough.

Answer 2

The third option is the most logical.

In regards to FSW missing some file events, I implemented this: 1) FSW Object which fires on FileCreate 2) tmrFileCheck, ticks = 5000 (5 seconds) - Calls tmrFileChec_Tick

When the FileCreate event occurs, if (tmrFileCheck.Enabled == false) then tmrFileCheck.Start()

This way, after 10 seconds tmrFileCheck_Tick fires which a) tmrFileCheck.Stop() b) CheckForStragglerFiles

Of tests I've run, this works effectively where there are a < 100 files created per minute.

A variant is to merely have a timer tick ever NN seconds and sweep the directory(ies) for straggler files.

Another variant is to hire me to press F5 to refresh the window and call you when there are straggler files; just a suggestion. :-P