Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Best way to save 10 GB vector to disk in Haskell

Tags:

haskell

I have a 10GB Data.Vector.Unboxed vector that I want to efficiently save to disk. What's the best, most efficient way? I plan to read it from a memory-mapped file too.

I have seen this package this package but only works with Storable but I need to stay with unboxed.

I was thinking of converting to a list but I am assuming this is not very ideal.

like image 639
jap Avatar asked Apr 23 '14 14:04

jap


1 Answers

You can convert between Vector types at the cost of an O(n) traversal of the entire vector. The function you're looking for is convert. As long as you're not planning to write this vector out to disk often, this cost should not be significant over all, and certainly faster than actually writing the vector out to disk. However, if you find yourself paying this cost often, you should probably rethink the algorithm.

like image 153
cassandracomar Avatar answered Sep 18 '22 15:09

cassandracomar