GHC can't handle large lookup tables

Question

I need to use a lookup table in 3 dimensions. The table itself has 73x73x73 (389017) double values.

module Hammer.Texture.Table3D where

import qualified Data.Vector         as V
import qualified Data.Vector.Unboxed as U

import           Data.Vector.Unboxed (Vector)

table3D :: V.Vector (V.Vector (Vector Double))
table3D = V.fromList [table0, table1, ... table72]

table0 = V.fromList $ map U.fromList [
  [1.973921e+01, 0.000000e+00, ... 0.000000e+00],
  .....
  [1.973921e+01, 0.000000e+00, ... 0.000000e+00]]
.....
table72 = V.fromList $ map U.fromList [
  [1.973921e+01, 0.000000e+00, ... 0.000000e+00],
  .....
  [1.973921e+01, 0.000000e+00, ... 0.000000e+00]]

The problem is that GHC can't handle this size of Vector Double or [Double], the GHC compilation takes a lot of time (~ 2 min) until, finally, the memory blows up. It seems there is a memory leak on GHC or some bug because it works fine for very large String ([Char]).

What solutions, if any, are available for creating a "large" lookup tables (Double type) using GHC?

Answer 1

I can think of two possibilities:

1. Serialize the vectors to a file, and at program startup deserialize them (perhaps using unsafeInterleaveIO if you would like this to happen when the lookup table is first used rather than when the program starts).
2. If the lookup table is really as sparse as your pseudocode suggests, consider using a sparse data structure -- e.g. Data.Map or even just a plain function. If necessary, you can use this sparse data structure to generate your vectors (again at runtime).