Error: reached the recursion limit while instantiating `func::<[closure]>`

Question

I am trying to test if a binary search tree is valid:

use std::{cell::RefCell, rc::Rc};

pub struct TreeNode {
    val: i32,
    left: Option<Rc<RefCell<TreeNode>>>,
    right: Option<Rc<RefCell<TreeNode>>>,
}

pub fn is_valid_bst(root: Option<Rc<RefCell<TreeNode>>>) -> bool {
    preorder_traverse(root.as_ref(), |_| true)
}

fn preorder_traverse<F: Fn(i32) -> bool>(root: Option<&Rc<RefCell<TreeNode>>>, predict: F) -> bool {
    if let Some(node) = root {
        let root_val = root.as_ref().unwrap().borrow().val;
        if !predict(root_val) {
            return false;
        }
        preorder_traverse(node.borrow().left.as_ref(), |v| v < root_val)
            && preorder_traverse(node.borrow().right.as_ref(), |v| v > root_val)
    } else {
        true
    }
}

(Playground):

This code triggers the following error message and that seems non-sense to me:

error: reached the recursion limit while instantiating `preorder_traverse::<[closure@src/lib.rs:19:56: 19:72 root_val:&i32]>`
  --> src/lib.rs:13:1
   |
13 | / fn preorder_traverse<F: Fn(i32) -> bool>(root: Option<&Rc<RefCell<TreeNode>>>, predict: F) -> bool {
14 | |     if let Some(node) = root {
15 | |         let root_val = root.as_ref().unwrap().borrow().val;
16 | |         if !predict(root_val) {
...  |
23 | |     }
24 | | }
   | |_^

I've found a potentially related Rust issue, but it seems outdated and I cannot understand the quoted message in the original issue well.

• What hit the recursion limit?
• How can I work around this if I want to encapsulate the predication logic in a closure or something else?

The algorithm to validate binary search tree in this code is not correct, but I still think the original code should compile.

Answer 1

@Lukas Kalbertodt provides a simpler example, which I'll use as a basis for the explanation:

fn foo<F: Fn()>(x: bool, _: F) {
    if x {
        foo(false, || {}) // line 3
    }
}

fn main() {
    foo(true, || {}); // line 8
}

The important point here is that each closure has a unique type, so let's instantiate this program:

• 1st closure, in main, let's name the type main#8.
• 1st instantiation of foo, in main, foo<[main#8]>.
• 2nd closure, in foo, let's name the type {foo<[main#8]>}#3.
• 2nd instantiation of foo, in foo, foo<[{foo<[main#8]>}#3]>.
• 3rd closure, in foo, let's name type {foo<[{foo<[main#8]>}#3]>}#3.
• 3rd instantiation of foo, in foo, foo<[{foo<[{foo<[main#8]>}#3]>}#3]>.
• ...

Each new instantiation of foo creates a new closure type, each new closure type creates a new instantiation of foo, this is a recursion without a base case: stack overflow.

---

You can solve the problem by NOT creating a closure when recursively calling preorder_traverse:

• either using type erasure, although there's a runtime overhead,
• or simply by using a separate inner function for recursion, since it's independent of F.

Example:

fn preorder_traverse_impl(
    root: Option<&Rc<RefCell<TreeNode>>>,
    parent_value: i32,
    predict: fn(i32, i32) -> bool
)
    -> bool
{
    if let Some(node) = root {
        let root_val = root.as_ref().unwrap().borrow().val;
        if !predict(root_val, parent_value) {
            return false;
        }
        preorder_traverse_impl(node.borrow().left.as_ref(), root_val, lessThan)
            && preorder_traverse_impl(node.borrow().right.as_ref(), root_val, greaterThan)
    } else {
        true
    }
}

fn preorder_traverse<F: Fn(i32) -> bool>(root: Option<&Rc<RefCell<TreeNode>>>, predict: F) -> bool {
    if let Some(node) = root {
        let root_val = root.as_ref().unwrap().borrow().val;
        if !predict(root_val) {
            return false;
        }
        preorder_traverse_impl(node.borrow().left.as_ref(), root_val, lessThan)
            && preorder_traverse_impl(node.borrow().right.as_ref(), root_val, greaterThan)
    } else {
        true
    }
}

On nightly you could also create a predicate type and implement Fn for it (LessThan<i32> and GreaterThan<i32>).