FileStream and memory usage

Question

I've written the following program which purpose is to create a file of a give size with some random data in it. The program works fine and does what it's suppose to do. However, I don't understand why it consumes 5GB of RAM (see screenshot of my Task Manager). While I am writing the file with random data, I am not creating new objects. What am I missing? I would expect this program to take no memory at all.

The big problem I have right now is that in the middle on the file generation, the machine is dying...

class Program
{
    static void Main(string[] args)
    {
        CreateFile("test.dat", 10 * 1024 * 1024);
    }

    public static void CreateFile(string path, long approximativeFileSizeInKb)
    {
        RandomNumberGenerator randomNumber = RandomNumberGenerator.Create();

        byte[] randomData = new byte[64 * 1024];

        int numberOfIteration = 0;
        randomNumber.GetNonZeroBytes(randomData);

        using (FileStream fs = File.Create(path, 64 * 1024))
        {
            while (numberOfIteration++ * 64 < approximativeFileSizeInKb)
            {
                fs.Write(randomData, 0, randomData.Length);
            }
        }
    }
}

Answer 1

Change your line which reads:

using (FileStream fs = File.Create(path, 64 * 1024))

to

using (FileStream fs = File.Create(path, 64 * 1024, FileOptions.WriteThrough))

and see how that does for you.

Answer 2

File system writes are always buffered by the OS.

You are calling FileSystem.Write faster than your hardware can handle the writes, thus the OS is caching all of your writes.

Even if you called FileSystem.Flush, you would still be writing faster than your hardware can handle the writes.

Get a faster hard disk subsystem. Preferrable an RAID controller with lots of on board memory connected to a large RAID 5 or 6 array with server based hard drives with 64MB caches set with write buffering.

(To alleviate this behavior add the flag FileOptions.WriteThrough to your File.Create call.)