Binary file format with 1000s of records in C#

Question

I would like to have an array model objects to be serialized to a binary stream. The model class will mainly have string and integer properties.

I believe that I can mark the class as [Serializable] and use the binary formattter, however I'd be interested to know whether you think this is the best way bearing in mind that my priority is to have as smaller file as possible for transfer over a low bandwidth connection (I can zip/unzip the file too).

The file could have 1000s of records, so ideally I'd like to be able to append to disk and read from disk record by record, without ever having to have the entire file in memory at once.

So my priorities are: small file size and efficient memory use.

Maybe there is a pre-written framework for this? It seems easy to do with XML and CSV files! Hopefully it is with a custom binary format too.

thanks

Answer 1

I suggest protobuf.net which is very efficient.

Having said that, this will not be able to handle serialising/deserialsing individual objects in your collection. That part you need to implement yourself.

• One solution is to: Store objects as individual files in a folder. File name will contain a reference so that based on name, you can find the object you need.

• Another is to have one file but keep an index file which keeps a list of all objects and their positions in the file. This is a lot more complicated as when you are saving an object which is in the middle of the file, you have to move all other addresses, and perhaps a b-tree is more effective.

Answer 2

Another option is to just serialize to a fixed-width text file format and let ZIP handle the compression. Fixed-width means you can easily use a MemoryMappedFile to walk through each record without needing to load the entire file into memory.