Multiple FileStreams for reading/writing different sections of same file concurrently in independent threads

Question

I have a large disk file (around 8 GB) containing several million records that I need to read, process in memory, and write back to another file. All the records are of a fixed length (say, 100 bytes).

I was thinking of parallelizing my process to run on multiple threads (typically 4–8), each of which would be (exclusively) assigned a particular section of the file to process (for example, 1 GB chunks). Since each thread would restrict its reads and writes to the section of the file it has been assigned, there is no risk of race hazards from my code.

Am I allowed to initialize multiple threads, each with its own FileStream, for reading from / writing to the same file without locking, without risking corruption? Assume that the target file has been expanded to its full size in advance (using FileStream.SetLength), and that the appropriate FileShare flag is specified when opening each FileStream.

Also, would I risk incurring slow-downs due to loss of buffering if multiple threads access the same file simultaneously? I am concerned about the “Detection of stream position changes” section in the MSDN documentation on the FileStream class, which states:

When a FileStream object does not have an exclusive hold on its handle, another thread could access the file handle concurrently and change the position of the operating system's file pointer that is associated with the file handle. […]

If an unexpected change in the handle position is detected in a call to the Read method, the .NET Framework discards the contents of the buffer and reads the stream from the file again. This can affect performance, depending on the size of the file and any other processes that could affect the position of the file stream.

Would this apply in my case, or are the file handles created by FileStream instances distinct and independent, even if accessing the same file?

Answer 1

This is perfectly safe.

There is no risk of the problem mentioned in the MSDN article as it only applies if you make changes to underlying handle yourself. You are not accessing the handle at all.

You will notice random disk IO though which can destroy performance. You probably want to mitigate this by reading big chunks from the file (16MB or so) and using a lock to prevent concurrent Read and Write calls. Notice that you need to prevent concurrent calls even on different FileStream instances because IOs are not treated atomically by the OS. Internally they get split to small sizes to allow for fairness and predictable latency. This leads to random IO.

Why don't you just create one reader thread pushing buffers into a BlockingCollection? You can process that collection using Parallel.ForEach on multiple threads.