What is the best way to read a huge file (around 1 TB) in haskell. Basically the file contains a matrix of integer data. and I may need to (efficiently ) calculate the correlation between the different rows or between columns.
I have previously used pytables for this but was thinking of trying the same in haskell. I know haskell has some hdf5 bindings but is there any other options which I am not aware of ?
As in any other language: you seek (using System.IO.hSeek
), and then use binary IO (Data.ByteString.hGet
). Then you parse the result (e.g. using attoparsec) and process as needed.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With