Stack Overflow in C function call - MS Visual C++ 2010 Express

Question

I have written a function in C, which, when called, immediately results in a stack overflow.

Prototype: void dumpOutput( Settings *, char **, FILE * );

Calling line: dumpOutput( stSettings, sInput, fpOut );

At the time of calling it, stSettings is already a pointer to Settings structure, sInput is a dynamically allocated 2D array and fpOut is a FILE *. It reaches all the way to the calling line without any errors, no memory leaks etc.

The actual function is rather lengthy and i think its not worth sharing it here as the overflow occurs just as the code enters the function (called the prologue part, i think)

I have tried calling the same function directly from main() with dummy variables for checking if there are any problems with passed arguments but it still throws the stack overflow condition.

The error arises from the chkstk.asm when the function is called. This asm file (according to the comments present in it) tries to probe the stack to check / allocate the memory for the called function. It just keeps jumping to Find next lower page and probe part till the stack overflow occurs.

The local variables in dumpOutput are not memory beasts either, just 6 integers and 2 pointers.

The memory used by code at the point of entering this function is 60,936K, which increases to 61,940K at the point when the stack overflow occurs. Most of this memory goes into the sInput. Is this the cause of error? I don't think so, because only its pointer is being passed. Secondly, i fail to understand why dumpOutput is trying to allocate 1004K of memory on stack?

I am totally at a loss here. Any help will be highly appreciated.

Thanks in advance.

Answer 1

By design, it is _chkstk()'s job to generate a stack overflow exception. You can diagnose it by looking at the generated machine code. After you step into the function, right-click the edit window and click Go To Disassembly. You ought to see something similar to this:

003013B0  push        ebp  
003013B1  mov         ebp,esp 
003013B3  mov         eax,1000D4h                  ; <== here
003013B8  call        @ILT+70(__chkstk) (30104Bh) 

The value passed through the EAX register is the important one, that's the amount of stack space your function needs. Chkstk then verifies it is actually available by probing the pages of stack. If you see it repeatedly looping then the value for EAX in your code is high. Like mine, it is guaranteed to consume all bytes of the stack. And more. Which is what it protects against, you normally get an access violation exception. But there's no guarantee, your code may accidentally write to a mapped page that belongs to, say, the heap. Which would produce an incredibly difficult to diagnose bug. Chkstk() helps you find these bugs before you blow your brains out in frustration.

I simply did it with this little test function:

void test()
{
    char kaboom[1024*1024];
}

We can't see yours, but the exception says that you either have a large array as a local variable or you are passing a large value to _alloca(). Fix by allocating that array from the heap instead.