Out of memory exception for a matrix

Question

I have the "'System.OutOfMemoryException" exception for this simple code (a 10 000 * 10 000 matrix) multiplied by itself:

#time

#r "Microsoft.Office.Interop.Excel"
#r "FSharp.PowerPack.dll"

open System
open System.IO

open Microsoft.FSharp.Math
open System.Collections.Generic


let mutable Matrix1 = Matrix.create 10000 10000 0.

let matrix4 = Matrix1 * Matrix1

I have the following error:

System.OutOfMemoryException: An exception 'System.OutOfMemoryException' has been raised
   Microsoft.FSharp.Collections.Array2DModule.ZeroCreate[T](Int32 length1, Int32 length2)
   Microsoft.FSharp.Math.DoubleImpl.mulDenseMatrixDS(DenseMatrix`1 a, DenseMatrix`1 b)
   Microsoft.FSharp.Math.SpecializedGenericImpl.mulM[a](Matrix`1 a, Matrix`1 b)
   <StartupCode$FSI_0004>.$FSI_0004.main@() dans C:\Users\XXXXXXX\documents\visual studio 2010\Projects\Library1\Library1\Module1.fs:line 92
Stop due to an error

I have therefore 2 questions:

1. I have a 8 GB memory on my computer and according to my calculation a 10 000 * 10 000 matrix should take 381 MB [computed this way : 10 000 * 10 000 = 100 000 000 integers in the matrix => 100 000 000 * 4 bytes (integers of 32 bits) = 400 000 000 => 400 000 000 / (1024*1024) = 381 MB] so I cannot understand why there is an OutOfMemoryException

2. More generally (it's not the case here I think), I have the impression that F# interactive registers all the data and therefore overloads the memory, do you know of a way to free all the data registered by F# interactive without exiting F#?

Answer 1

In summary, fsi is a 32-bit process; at most it can hold 2GB of data. Run your test as a 64-bit Windows application; you can increase the size of the matrix, but it still has 2GB limit of .NET objects.

I correct your calculation a little bit. Matrix1 is a float matrix, so each element occupies 8 bytes in memory. The total size of Matrix1 and matrix4 in memory is at least:

2 * 10000 * 10000 * 8 = 1 600 000 000 bytes ~ 1.6 GB

(ignoring some bookkeeping parts of matrix)

So it's no surprise when fsi*32 runs out of memory in this case.

Execute the test as a 64-bit Windows process, you can create float matrices of size around 15000 but not more than that. Check out this informative article for concrete numbers with different types of matrix elements.