Read from a huge MemoryStream in C#

Question

I use a BinaryReader (MemoryStream(MyByteArray)) to read variable sized records and process them all in memory. This works well as long as my bytestream, which is in the array, is less than about 1.7 GB in size. After that (which is the maximum size of an integer in my 64-bit system) you cannot create a larger bytearray, although I have enough real memory. So my solution has been to read the bytestream and split it into several byte arrays.

Now however, I cannot "read" across the byte array boundaries, and, as my data is in a variable format, I cannot ensure that byte arrays always finish on a whole record.

This must be a common problem for people processing very large datasets and still have the need for speed.

How do I handle this problem?

Answer 1

Edit: Reading up on the basics, I realize that memory-mapped files might be slower than normal I/O for sequential access.

Have you tried something like this:

var stream = new FileStream("data", 
    FileMode.Open, 
    FileAccess.Read, 
    FileShare.Read, 
    16 * 1024, 
    FileOptions.SequentialScan)

var reader = new BinaryReader(stream);

If your data resides in a file and you can use .NET 4.0 consider using MemoryMappedFile.

You can then either use a MemoryMappedViewStream to get a stream or use a MemoryMappedViewAccessor to get a BinaryReader-like interface.