How to persist large data for efficient deserialization in Haskell

Question

I'm facing the general problem to compile large data sets to an on-disk representation, which can be efficiently de-serialized to native in-memory Haskell data structures.

More specifically, I have a large amount of graph-data with various attributes associated with edges and vertices. In C/C++ I have compiled the data to a mmap()able represenation for maximum efficiency, which currently results in about 200MiB worth of C-structures (and whose text representation is about 600 MiB).

What is the next-best thing I can do in (GHC) Haskell?

Answer 1

Use the package binary. It provides a toolbox to efficiently serialize and deserialize data in Haskell. binary can automagically derive instances of the requred typeclasses for you, but you can also write optimized instances manually.

Quoted from the originial description page:

The binary package

Efficient, pure binary serialisation using lazy ByteStrings. Haskell values may be encoded to and from binary formats, written to disk as binary, or sent over the network. Serialisation speeds of over 1 G/sec have been observed, so this library should be suitable for high performance scenarios.