Scala can't deal with huge objects?

Question

I have a syntactically correct Scala source file holding a huge data structure. The code looks like this:

object ChiSquareAlpha
{
  val ChiSquareToAlphaIndexTable = Map(
    1 -> Array(0.00016,0.00063,0.00393,0.01579,0.06418,0.14847,0.45494,1.07419,1.64237,2.70554,3.84146,5.41189,6.63490,10.82757),
    .. 9998 other entries ..
    10000 -> Array(9673.95,9711.71,9768.53,9819.19,9880.79,9925.36,9999.33,10073.68,10118.82,10181.66,10233.75,10292.58,10331.93,10442.73)
  )
}

So you can see, the object holds a huge lookup table, a Map[Int, Array[Double]] with 10000 entries. Compilation takes about a minute, but finishes without warnings or errors. But at runtime it fails with:

java.lang.ClassFormatError: Invalid this class index 4280 in constant pool in class file bm/statistic/ChiSquareAlpha$
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.scala$tools$nsc$util$ScalaClassLoader$$super$findClass(ScalaClassLoader.scala:88)
at scala.tools.nsc.util.ScalaClassLoader$class.findClass(ScalaClassLoader.scala:44)
at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.findClass(ScalaClassLoader.scala:88)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.scala$tools$nsc$util$ScalaClassLoader$$super$loadClass(ScalaClassLoader.scala:88)
at scala.tools.nsc.util.ScalaClassLoader$class.loadClass(ScalaClassLoader.scala:50)
at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.loadClass(ScalaClassLoader.scala:88)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at bm.statistic.ChiSquareTest.testOfIndependence(ChiSquareTest.scala:323)
at ...

I tested this with scala 2.9.2 and scala 2.10. Same result. When the object is much smaller - let's say 500 map entries - the program works. So I guess scala is unable to deal with such big files. Any solution proposals?

Answer 1

You are correct, your syntax looks fine.

I apologize for the disappointing answer, this is an open bug with Scala:

Scala Programming Language SI-6543 scala compile fails on simple but lengthy repetitive script (ClassNotFound and/or StackOverflowError)

It's fix priority is Major (which is high) and you can find more information about it (they explain why it happens).

You can keep track of the progress. Meanwhile, you can create that information in a flat file and read the data from it as a temporary fix.