Using BinaryWriter or BinaryReader in async code

Question

I have a list of float to write to a file. The code below does the thing but it is synchronous.

List<float> samples = GetSamples();

using (FileStream stream = File.OpenWrite("somefile.bin"))
using (BinaryWriter binaryWriter = new BinaryWriter(stream, Encoding.Default, true))
{
    foreach (var sample in samples)
    {
        binaryWriter.Write(sample);
    }
}

I want to do the operation asynchronously but the BinaryWriter does not support async operations, which is normal since it just only writes a few bytes each time. But most of the time the operation uses file I/O and I think it can and should be asynchronous.

I tried to write to a MemoryStream with the BinaryWriter and when that finished I copied the MemoryStream to the FileStream with CopyToAsync, however this caused a performance degradation (total time) up to 100% with big files.

How can I convert the whole operation to asynchronous?

Answer 1

Normal write operations usually end up being completed asynchronously anyway. The OS accepts writes immediately into the write cache, and flushes it to disk at some later time. Your application isn't blocked by the actual disk writes.

Of course, if you are writing to a removable drive then write cache is typically disabled and your program will be blocked.

---

I will recommend that you can dramatically reduce the number of operations by transferring a large block at a time. To wit:

1. Allocate a new T[BlockSize] of your desired block size.
2. Allocate a new byte[BlockSize * sizeof (T)]
3. Use List<T>.CopyTo(index, buffer, 0, buffer.Length) to copy a batch out of the list.
4. Use Buffer.BlockCopy to get the data into the byte[].
5. Write the byte[] to your stream in a single operation.
6. Repeat 3-5 until you reach the end of the list. Careful about the final batch, which may be a partial block.

Answer 2

Your memory stream approach makes sense, just make sure to write in batches rather than waiting for the memory stream to grow to the full size of the file and then writing it all at once.

Something like this should work fine:

var data = new float[10 * 1024];
var helperBuffer = new byte[4096];

using (var fs = File.Create(@"D:\Temp.bin"))
using (var ms = new MemoryStream(4096))
using (var bw = new BinaryWriter(ms))
{
  var iteration = 0;

  foreach (var sample in data)
  {
    bw.Write(sample);

    iteration++;

    if (iteration == 1024)
    {
      iteration = 0;
      ms.Position = 0;

      ms.Read(helperBuffer, 0, 1024 * 4);
      await fs.WriteAsync(helperBuffer, 0, 1024 * 4).ConfigureAwait(false);
    }
  }
}

This is just sample code - make sure to handle errors properly etc.

Answer 3

Sometimes, these helper classes are anything but helpful.

Try this:

List<float> samples = GetSamples();

using (FileStream stream = File.OpenWrite("somefile.bin"))
{
    foreach (var sample in samples)
    {
        await stream.WriteAsync(BitConverter.GetBytes(sample), 0, 4);
    }
}