parse content away from structure in a binary file

Question

Using C#, I need to read a packed binary file created using FORTRAN. The file is stored in an "Unformatted Sequential" format as described here (about half-way down the page in the "Unformatted Sequential Files" section):

http://www.tacc.utexas.edu/services/userguides/intel8/fc/f_ug1/pggfmsp.htm

As you can see from the URL, the file is organized into "chunks" of 130 bytes or less and includes 2 length bytes (inserted by the FORTRAN compiler) surrounding each chunk.

So, I need to find an efficient way to parse the actual file payload away from the compiler-inserted formatting.

Once I've extracted the actual payload from the file, I'll then need to parse it up into its varying data types. That'll be the next exercise.

My first thoughts are to slurp up the entire file into a byte array using File.ReadAllBytes. Then, just iterate through the bytes, skipping the formatting and transferring the actual data to a second byte array.

In the end, that second byte array should contain the actual file contents minus all the formatting, which I'd then need to go back through to get what I need.

As I'm fairly new to C#, I thought there might be a better, more accepted way of tackling this.

Also, in case it's helpful, these files could be fairly large (say 30MB), though most will be much smaller...

Answer 1

One way to read files like this is record by record (e.g., read the length bytes and then the data chunk, building up a list of records, which are just byte arrays). The collection of records is then passed to further parsing routines.

However, if you're on 4.0, there is a new class for file mapping which would be more efficient yet work similarly to ReadAllBytes.

If you're using ReadAllBytes or MemoryMappedFile it's nice to build an in-memory "index" into the large binary file by parsing all the record lengths first. This is especially useful if you will only need certain records.