Writing to memory mapped file is slower than non-memory mapped file

Question

I am trying to use a memory mapped file to write an application that has high IO demands. In this application, I have burst of data that is received at a rate faster than what the disk is able to support. In order to avoid buffering logic in my application, I thought about using memory mapped file. With this kind of file, I would simply write in memory that is mapped to a file (faster than what the disks can support) and the OS would eventually flush this data to disk. The OS is therefore doing the buffering for me.

After experiment, I see that memory mapped files makes it faster to write in memory but the flushing to disk is slower than with a normal file. Here is what leads me to that conclusion. Here is a piece of code that simply writes as fast as it can to a non-memory mapped file:

    private static void WriteNonMemoryMappedFile(long fileSize, byte[] bufferToWrite)
    {
        Console.WriteLine(" ==> Non memory mapped file");

        string normalFileName = Path.Combine(Path.GetTempPath(), "MemoryMappedFileWriteTest-NonMmf.bin");
        if (File.Exists(normalFileName))
        {
            File.Delete(normalFileName);
        }

        var stopWatch = Stopwatch.StartNew();
        using (var file = File.OpenWrite(normalFileName))
        {
            var numberOfPages = fileSize/bufferToWrite.Length;

            for (int page = 0; page < numberOfPages; page++)
            {
                file.Write(bufferToWrite, 0, bufferToWrite.Length);
            }
        }

        Console.WriteLine("Non-memory mapped file is now closed after {0} seconds ({1} MB/s)", stopWatch.Elapsed.TotalSeconds, GetSpeed(fileSize, stopWatch));
    }

This code results in this:

==> Non memory mapped file
Non-memory mapped file is now closed after 10.5918587 seconds (966.687541390441 MB/s)

As you can see, my disks are quite fast. This will be my benchmark for memory mapped files.

Now I tried to write the same data to a memory mapped file using unsafe code (because this is what I intend to do in my application):

    [DllImport("msvcrt.dll", EntryPoint = "memcpy", CallingConvention = CallingConvention.Cdecl, SetLastError = false)]
    public static extern IntPtr memcpy(IntPtr dest, IntPtr src, UIntPtr count);

    private static unsafe void WriteMemoryMappedFileWithUnsafeCode(long fileSize, byte[] bufferToWrite)
    {
        Console.WriteLine(" ==> Memory mapped file with unsafe code");

        string fileName = Path.Combine(Path.GetTempPath(), "MemoryMappedFileWriteTest-MmfUnsafeCode.bin");
        if (File.Exists(fileName))
        {
            File.Delete(fileName);
        }

        string mapName = Guid.NewGuid().ToString();

        var stopWatch = Stopwatch.StartNew();
        using (var memoryMappedFile = MemoryMappedFile.CreateFromFile(fileName, FileMode.Create, mapName, fileSize, MemoryMappedFileAccess.ReadWrite))
        using (var view = memoryMappedFile.CreateViewAccessor(0, fileSize, MemoryMappedFileAccess.Write))
        {
            unsafe
            {
                fixed (byte* pageToWritePointer = bufferToWrite)
                {
                    byte* pointer = null;
                    try
                    {
                        view.SafeMemoryMappedViewHandle.AcquirePointer(ref pointer);

                        var writePointer = pointer;

                        var numberOfPages = fileSize/bufferToWrite.Length;

                        for (int page = 0; page < numberOfPages; page++)
                        {
                            memcpy((IntPtr) writePointer, (IntPtr) pageToWritePointer, (UIntPtr) bufferToWrite.Length);
                            writePointer += bufferToWrite.Length;
                        }
                    }
                    finally
                    {
                        if (pointer != null)
                            view.SafeMemoryMappedViewHandle.ReleasePointer();
                    }
                }
            }

            Console.WriteLine("All bytes written in MMF after {0} seconds ({1} MB/s). Will now close MMF. This may be long since some content may not have been flushed to disk yet.", stopWatch.Elapsed.TotalSeconds, GetSpeed(fileSize, stopWatch));
        }

        Console.WriteLine("File is now closed after {0} seconds ({1} MB/s)", stopWatch.Elapsed.TotalSeconds, GetSpeed(fileSize, stopWatch));
    }

I then get this:

==> Memory mapped file with unsafe code
All bytes written in MMF after 6.5442406 seconds (1564.73302033172 MB/s). Will now close MMF. This may be long since some content may not have been flushed to disk yet.
File is now closed after 18.8873186 seconds (542.162704287661 MB/s)

As you can see, this is much slower. It writes at about 56% of a non memory mapped file.

I then tried another thing. I tried to use a ViewStreamAccessor instead of unsafe code:

    private static unsafe void WriteMemoryMappedFileWithViewStream(long fileSize, byte[] bufferToWrite)
    {
        Console.WriteLine(" ==> Memory mapped file with view stream");
        string fileName = Path.Combine(Path.GetTempPath(), "MemoryMappedFileWriteTest-MmfViewStream.bin");
        if (File.Exists(fileName))
        {
            File.Delete(fileName);
        }

        string mapName = Guid.NewGuid().ToString();

        var stopWatch = Stopwatch.StartNew();
        using (var memoryMappedFile = MemoryMappedFile.CreateFromFile(fileName, FileMode.Create, mapName, fileSize, MemoryMappedFileAccess.ReadWrite))
        using (var viewStream = memoryMappedFile.CreateViewStream(0, fileSize, MemoryMappedFileAccess.Write))
        {
            var numberOfPages = fileSize / bufferToWrite.Length;

            for (int page = 0; page < numberOfPages; page++)
            {
                viewStream.Write(bufferToWrite, 0, bufferToWrite.Length);
            }                

            Console.WriteLine("All bytes written in MMF after {0} seconds ({1} MB/s). Will now close MMF. This may be long since some content may not have been flushed to disk yet.", stopWatch.Elapsed.TotalSeconds, GetSpeed(fileSize, stopWatch));
        }

        Console.WriteLine("File is now closed after {0} seconds ({1} MB/s)", stopWatch.Elapsed.TotalSeconds, GetSpeed(fileSize, stopWatch));
    }

I then get this:

==> Memory mapped file with view stream
All bytes written in MMF after 4.6713875 seconds (2192.06548076352 MB/s). Will now close MMF. This may be long since some content may not have been flushed to disk yet.
File is now closed after 16.8921666 seconds (606.198141569359 MB/s)

Once again, this is significantly slower than with a non memory mapped file.

So, does anyone knows how to make memory mapped files as fast as non memory mapped files when writing?

By the way, here is the remainder of my test program:

    static void Main(string[] args)
    {
        var bufferToWrite = Enumerable.Range(0, Environment.SystemPageSize * 256).Select(i => (byte)i).ToArray();
        long fileSize = 10 * 1024 * 1024 * 1024L; // 2 GB

        WriteNonMemoryMappedFile(fileSize, bufferToWrite);
        WriteMemoryMappedFileWithUnsafeCode(fileSize, bufferToWrite);
        WriteMemoryMappedFileWithViewStream(fileSize, bufferToWrite);
    }

    private static double GetSpeed(long fileSize, Stopwatch stopwatch)
    {
        var mb = fileSize / 1024.0 / 1024.0;
        var mbPerSec = mb / stopwatch.Elapsed.TotalSeconds;
        return mbPerSec;
    }

EDIT 1:

As suggested by usr, I tried to use the SequenctialScan option. Unfortunately, it didn't have any impact. Here is the change that I did:

        using (var file = new FileStream(fileName, FileMode.CreateNew, FileAccess.ReadWrite, FileShare.None, 4096, FileOptions.SequentialScan))
        using (var memoryMappedFile = MemoryMappedFile.CreateFromFile(file, mapName, fileSize, MemoryMappedFileAccess.ReadWrite, null, HandleInheritability.None, leaveOpen: false))

Answer 1

From the SDK documentation:

Modified pages in the unmapped view are not written to disk until their share count reaches zero, or in other words, until they are unmapped or trimmed from the working sets of all processes that share the pages. Even then, the modified pages are written "lazily" to disk; that is, modifications may be cached in memory and written to disk at a later time. To minimize the risk of data loss in the event of a power failure or a system crash, applications should explicitly flush modified pages using the FlushViewOfFile function.

The .NET programmers took the last sentence seriously, the MemoryMappedViewStream.Dispose() method you are calling does in fact call FlushViewOfFile(). That takes time, you are seeing that back in your profile results. It is technically possible to bypass this call, don't call Dispose() and let the finalizer close the view handle.

FileStream does not do the equivalent for a file (FlushFileBuffers) so you get the full benefit of the lazy write from the file system cache to the disk. Happens long after the Dispose() call, unobservable to your program.