.NetCore - FileSystemWatcher on a network drive, unsafe code Win32 API crash

Question

I have a small Dotnet core program (3.1.8), with some FileWatchers. They watch folders on a network drive. With some load (200 - 250 files maximum here), the program crashes unexpectedly. These files come at the same time, moved by another process on another server thanks to a Biztalk app, I don't think it's relevant here but I wanted to mention it.

The filewatchers initialization:

private void InitializeInnerFilewatcher(List<string> filters)
{
        _watcher = new FileSystemWatcher(WatchPath);
        _watcher.InternalBufferSize = 65536;
        if (filters.Count > 1)
        {
            _watcher.Filter = FILTER_ALL; // *.*
            _customFilters = filters;
        }
        else
            _watcher.Filter = filters.First();
        _watcher.NotifyFilter = NotifyFilters.LastWrite | NotifyFilters.FileName;
        _watcher.Changed += new FileSystemEventHandler(FileCreatedOrChanged);
        _watcher.Created += new FileSystemEventHandler(FileCreatedOrChanged);
        _watcher.Renamed += new RenamedEventHandler(FileRenamed);
        _watcher.Error += Watcher_Error;
        _watcher.EnableRaisingEvents = true;
}

And here we have, the "process" part for each event triggered by the filewatcher:

private void TryHandle(FileSystemEventArgs arg)
{
        if (!File.Exists(arg.FullPath))
            return;

        if (!_customFilters.Any() || _customFilters.Any(x => PatternMatcher.MatchPattern(x, arg.Name)))
            _memoryCache.AddOrGetExisting(arg.FullPath, arg, _cacheItemPolicy);
 }

I tried to avoid any real process on triggered file system events, so I push the file path in the memoryCache and later I send it to a ServiceBus queue for processing the file by any consumer.

All this stuff seem to work pretty fine during all day, no high CPU no high Memory during all day. We're already logging all the application metrics in ApplicationInsights.

It's a 'real' crash so we don't have any logs, only a poor event in the Event Viewer and a dump file.

Event viewer : Faultinq module name: coreclr.dll, version: 470020.41105, time stamp: Ox5f3397ec

We can see, thanks to dotnet-dump, the error catched in the dump file:

> clrstack
OS Thread Id: 0xfd4c (27)
        Child SP               IP Call Site
00000022D55BE150 00007ffccc46789f [FaultingExceptionFrame: 00000022d55be150]
00000022D55BE650 00007FFC6D7A49D4 System.IO.FileSystemWatcher.ParseEventBufferAndNotifyForEach(Byte[]) [/_/src/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Win32.cs @ 249]
00000022D55BE6F0 00007FFC6D7A48E6 System.IO.FileSystemWatcher.ReadDirectoryChangesCallback(UInt32, UInt32, System.Threading.NativeOverlapped*) [/_/src/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Win32.cs @ 242]
00000022D55BE750 00007FFC6D6F189C System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 201]
00000022D55BE7C0 00007FFC6D7359B5 System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*) [/_/src/System.Private.CoreLib/src/System/Threading/Overlapped.cs @ 59]
00000022D55BE8F0 00007ffccc336ba3 [GCFrame: 00000022d55be8f0]
00000022D55BEAB0 00007ffccc336ba3 [DebuggerU2MCatchHandlerFrame: 00000022d55beab0]
> pe
Exception object: 000001e580001198
Exception type:   System.ExecutionEngineException
Message:          <none>
InnerException:   <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 80131506

As you can see, the error seems to happen directly on the FileSystemWatcher, in the Win32 API. I can't reproduce it, and it happens only on our Production environment, so no need to tell you I'm in an "emergency mode".

WinDbg is maybe a bit more detailed

Answer 1

Just a quick update, because I'm still on the way to fix it.

I created a MS Support issue. After many try we just succeed to reproduce it. We had to "play" with the network and simulating some "disturbances". It seems that FileSystemWatcher events didn't been sent as it should be (It's sent by TCP protocol, SMB way). Our team is still working on finding how it can happen..

MS agreed that this shouldn't crash the FileSystemWatcher in some unsafe code no matter if there was a real network issue. So, they just made a PR to add some security around it.

I'm still following the PR but it should be fix in .Net 5 and backported in .Net Core 3.1(.9).

Thanks for the help.

Answer 2

This issue has been fixed in master (6.0) and backported to 5.0 and 3.1.