Fatal Python error: Cannot recover from stack overflow

Question

I read on internet similar problem, but none of the answer could help me. I have a function that for each line of data (data have around 2'000'000 rows) do something and then recall the same function with different parameter depending on what it has done. The issue is that after a while I get this error in the terminal: 'Fatal Python error: Cannot recover from stack overflow.'

It seams that the most frequent mistake causing this error is infinite loop, but I controlled and have no infinite loop. Hence, to me the issue it that 'sys.getrecursionlimit()' is set to 3000, which means that after 3000 call of the same function it will give me the error.

The first thing, is that I dont understand the difference between 'Fatal Python error: Cannot recover from stack overflow.' in the terminal, or a 'RecursionError: maximum recursion depth exceeded in comparison' in the jupyternotebook. Indeed, to me it can come from the same mistake (e.g. infinite loop).

When replacing my function by a simple one called 'test_', I have the following code:

import sys
print(sys.getrecursionlimit())

def test_(x,t):
    x = x+1
    if x<t:
        test_(x=x,t=t)

print(test_(0,2971)) # output: None
print(test_(0,2972)) # RecursionError: maximum recursion depth exceeded in comparison

3000

None

--------------------------------------------------------------------------- RecursionError Traceback (most recent call last) in () 8 9 print(test_(0,2971)) ---> 10 print(test_(0,2972))

in test_(x, t) 5 x = x+1 6 if x 7 test_(x=x,t=t) 8 9 print(test_(0,2971))

... last 1 frames repeated, from the frame below ...

in test_(x, t) 5 x = x+1 6 if x 7 test_(x=x,t=t) 8 9 print(test_(0,2971))

RecursionError: maximum recursion depth exceeded in comparison

To overcome this issue I adapted the function without loosing the 'continuity of the run', so that I can use batches:

for i in np.arange(0,9000,2000):
    test_(i,i+2000)

Would someone have nicer solution? Also, in general its a bad idea to do recursive function when we know that we have a lot of iteration to do? Also does anyone knows how I could print the recursiondeepth at each loop?

I am working on a Linux virtual environement, with jupyter notebook, on python 3.6 with anaconda.

Answer 1

Please check this question (it worked for me): How do I get the current depth of the Python interpreter stack?

your code based on that answer:

import sys
import inspect
print(sys.getrecursionlimit())

def test_(x, t):
    print(len(inspect.stack()))
    x = x + 1
    if x < t:
        test_(x=x, t=t)

print(test_(0, 7))

output:

22
23
24
25
26
27
28
None