Error: C stack usage is too close to the limit

Question

I'm attempting to run some fairly deep recursive code in R and it keeps giving me this error:

Error: C stack usage is too close to the limit

My output from CStack_info() is:

Cstack_info()
    size    current  direction eval_depth 
67108864       8120          1          2 

I have plenty of memory on my machine, I'm just trying to figure out how I can increase the CStack for R.

EDIT: Someone asked for a reproducible example. Here's some basic sample code that causes the problem. Running f(1,1) a few times you'll get the error. Note that I've already set --max-ppsize = 500000 and options(expressions=500000) so if you don't set those you might get an error about one of those two things instead. As you can see, the recursion can go pretty deep here and I've got no idea how to get it to work consistently. Thanks.

f <- function(root=1,lambda=1) {
    x <- c(0,1);
    prob <- c(1/(lambda+1),lambda/(lambda+1));
        repeat {
      if(root == 0) {
        break;
      }
      else {
        child <- sample(x,2,replace=TRUE,prob);
        if(child[1] == 0 && child[2] == 0) {
          break;
        }
        if(child[1] == 1) {
          child[1] <- f(root=child[1],lambda);
        }
        if(child[2] == 1 && child[1] == 0) {
          child[2] <- f(root=child[2],lambda);
        }
      }
      if(child[1] == 0 && child[2] == 0) {
        break;
      }
      if(child[1] == 1 || child[2] == 1) {
        root <- sample(x,1,replace=TRUE,prob);
      }
        }
    return(root)
}

Answer 1

The stack size is an operating system parameter, adjustable per-process (see setrlimit(2)). You can't adjust it from within R as far as I can tell, but you can adjust it from the shell before starting R, with the ulimit command. It works like this:

$ ulimit -s # print default
8192
$ R --slave -e 'Cstack_info()["size"]'
   size 
8388608

8388608 = 1024 * 8192; R is printing the same value as ulimit -s, but in bytes instead of kilobytes.

$ ulimit -s 16384 # enlarge stack limit to 16 megs
$ R --slave -e 'Cstack_info()["size"]'
    size 
16777216 

To make a permanent adjustment to this setting, add the ulimit command to your shell startup file, so it's executed every time you log in. I can't give more specific directions than that, because it depends on exactly which shell you have and stuff. I also don't know how to do it for logging into a graphical environment (which will be relevant if you're not running R inside a terminal window).

Answer 2

I suspect that, regardless of stack limit, you'll end up with recursions that are too deep. For instance, with lambda = Inf, f(1) leads to an immediate recursion, indefinitely. The depth of the recursion seems to be a random walk, with some probability r of going deeper, 1 - r of finishing the current recursion. By the time you've hit the stack limit, you've made a large number of steps 'deeper'. This implies that r > 1 / 2, and the very large majority of time you'll just continue to recurse.

Also, it seems like it is almost possible to derive an analytic or at least numerical solution even in the face of infinite recursion. One can define p as the probability that f(1) == 1, write implicit expressions for the 'child' states after a single iteration, and equate these with p, and solve. p can then be used as the chance of success in a single draw from a binomial distribution.

Answer 3

This error is not due to memory it is due to recursion. A function is calling itself. This isn't always obvious from examining the definition of only one function. To illustrate the point, here is a minimal example of 2 functions that call each other:

change_to_factor <- function(x){
  x <- change_to_character(x)
  as.factor(x)
} 

change_to_character <- function(x){
  x <- change_to_factor(x)
  as.character(x)
}

change_to_character("1")

Error: C stack usage 7971600 is too close to the limit

The functions will continue to call each other recursively and will theoretically never complete, even if you increase the limit it will still be exceeded. It is only checks within your system that prevent this from occurring indefinitely and consuming all of the compute resources of your machine. You need to alter the functions to ensure that they won't indefinitely call itself (or each other) recursively.

Answer 4

This happened to me for a completely different reason. I accidentally created a superlong string while combining two columns:

output_table_subset = mutate(big_data_frame,
     combined_table = paste0(first_part, second_part, col = "_"))

instead of

output_table_subset = mutate(big_data_frame,
     combined_table = paste0(first_part, second_part, sep = "_"))

Took me for ever to figure it out as I never expected the paste to have caused the problem.