How to increase the Java stack size?

Question

Hmm... it works for me and with far less than 999MB of stack:

> java -Xss4m Test
0

(Windows JDK 7, build 17.0-b05 client VM, and Linux JDK 6 - same version information as you posted)

I assume you calculated the "depth of 1024" by the recurring lines in the stack trace?

Obviously, the stack trace array length in Throwable seems to be limited to 1024. Try the following program:

public class Test {

    public static void main(String[] args) {

        try {
            System.out.println(fact(1 << 15));
        }
        catch (StackOverflowError e) {
            System.err.println("true recursion level was " + level);
            System.err.println("reported recursion level was " +
                               e.getStackTrace().length);
        }
    }

    private static int level = 0;
    public static long fact(int n) {
        level++;
        return n < 2 ? n : n * fact(n - 1);
    }
}

If you want to play with the thread stack size, you'll want to look at the -Xss option on the Hotspot JVM. It may be something different on non Hotspot VM's since the -X parameters to the JVM are distribution specific, IIRC.

On Hotspot, this looks like java -Xss16M if you want to make the size 16 megs.

Type java -X -help if you want to see all of the distribution specific JVM parameters you can pass in. I am not sure if this works the same on other JVMs, but it prints all of Hotspot specific parameters.

For what it's worth - I would recommend limiting your use of recursive methods in Java. It's not too great at optimizing them - for one the JVM doesn't support tail recursion (see Does the JVM prevent tail call optimizations?). Try refactoring your factorial code above to use a while loop instead of recursive method calls.

The only way to control the size of stack within process is start a new Thread. But you can also control by creating a self-calling sub Java process with the -Xss parameter.

public class TT {
    private static int level = 0;

    public static long fact(int n) {
        level++;
        return n < 2 ? n : n * fact(n - 1);
    }

    public static void main(String[] args) throws InterruptedException {
        Thread t = new Thread(null, null, "TT", 1000000) {
            @Override
            public void run() {
                try {
                    level = 0;
                    System.out.println(fact(1 << 15));
                } catch (StackOverflowError e) {
                    System.err.println("true recursion level was " + level);
                    System.err.println("reported recursion level was "
                            + e.getStackTrace().length);
                }
            }

        };
        t.start();
        t.join();
        try {
            level = 0;
            System.out.println(fact(1 << 15));
        } catch (StackOverflowError e) {
            System.err.println("true recursion level was " + level);
            System.err.println("reported recursion level was "
                    + e.getStackTrace().length);
        }
    }

}

Add this option

--driver-java-options -Xss512m

to your spark-submit command will fix this issue.

It is hard to give a sensible solution since you are keen to avoid all sane approaches. Refactoring one line of code is the senible solution.

Note: Using -Xss sets the stack size of every thread and is a very bad idea.

Another approach is byte code manipulation to change the code as follows;

public static long fact(int n) { 
    return n < 2 ? n : n > 127 ? 0 : n * fact(n - 1); 
}

given every answer for n > 127 is 0. This avoid changing the source code.

I did Anagram excersize, which is like Count Change problem but with 50 000 denominations (coins). I am not sure that it can be done iteratively, I do not care. I just know that the -xss option had no effect -- I always failed after 1024 stack frames (might be scala does bad job delivering to to java or printStackTrace limitation. I do not know). This is bad option, as explained anyway. You do not want all threads in to app to be monstrous. However, I did some experiments with new Thread (stack size). This works indeed,

  def measureStackDepth(ss: Long): Long = {
    var depth: Long = 0
      val thread: Thread = new Thread(null, new Runnable() {
        override def run() {
          try {
          def sum(n: Long): Long = {depth += 1; if (n== 0) 0 else sum(n-1) + 1}
          println("fact = " + sum(ss * 10))
          } catch {
            case e: StackOverflowError => // eat the exception, that is expected
          }
        }
      }, "deep stack for money exchange", ss)
      thread.start()
      thread.join()
    depth
  }                                               //> measureStackDepth: (ss: Long)Long


  for (ss <- (0 to 10)) println("ss = 10^" +  ss + " allows stack of size " -> measureStackDepth((scala.math.pow (10, ss)).toLong) )
                                                  //> fact = 10
                                                  //| (ss = 10^0 allows stack of size ,11)
                                                  //| fact = 100
                                                  //| (ss = 10^1 allows stack of size ,101)
                                                  //| fact = 1000
                                                  //| (ss = 10^2 allows stack of size ,1001)
                                                  //| fact = 10000
                                                  //| (ss = 10^3 allows stack of size ,10001)
                                                  //| (ss = 10^4 allows stack of size ,1336)
                                                  //| (ss = 10^5 allows stack of size ,5456)
                                                  //| (ss = 10^6 allows stack of size ,62736)
                                                  //| (ss = 10^7 allows stack of size ,623876)
                                                  //| (ss = 10^8 allows stack of size ,6247732)
                                                  //| (ss = 10^9 allows stack of size ,62498160)

You see that stack can grow exponentially deeper with exponentially more stack alloted to the thread.