I recently read that signed integer overflow in C and C++ causes undefined behavior:
If during the evaluation of an expression, the result is not mathematically defined or not in the range of representable values for its type, the behavior is undefined.
I am currently trying to understand the reason of the undefined behavior here. I thought undefined behavior occurs here because the integer starts manipulating the memory around itself when it gets too big to fit the underlying type.
So I decided to write a little test program in Visual Studio 2015 to test that theory with the following code:
#include <stdio.h> #include <limits.h> struct TestStruct { char pad1[50]; int testVal; char pad2[50]; }; int main() { TestStruct test; memset(&test, 0, sizeof(test)); for (test.testVal = 0; ; test.testVal++) { if (test.testVal == INT_MAX) printf("Overflowing\r\n"); } return 0; }
I used a structure here to prevent any protective matters of Visual Studio in debugging mode like the temporary padding of stack variables and so on. The endless loop should cause several overflows of test.testVal
, and it does indeed, though without any consequences other than the overflow itself.
I took a look at the memory dump while running the overflow tests with the following result (test.testVal
had a memory address of 0x001CFAFC
):
0x001CFAE5 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x001CFAFC 94 53 ca d8 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
As you see, the memory around the int that is continuously overflowing remained "undamaged". I tested this several times with similar output. Never was any memory around the overflowing int damaged.
What happens here? Why is there no damage done to the memory around the variable test.testVal
? How can this cause undefined behavior?
I am trying to understand my mistake and why there is no memory corruption done during an integer overflow.
Integer overflow leads to the execution of buffer overflow vulnerability which allows the attacker to gain shell and elevate his privileges once this vulnerability is exploited. The validation checks are actually disabled by the integer overflow vulnerability thus resulting in execution of buffer overflow.
In contrast, the C standard says that signed integer overflow leads to undefined behavior where a program can do anything, including dumping core or overrunning a buffer. The misbehavior can even precede the overflow. Such an overflow can occur during addition, subtraction, multiplication, division, and left shift.
So, in C/C++ programming, undefined behavior means when the program fails to compile, or it may execute incorrectly, either crashes or generates incorrect results, or when it may fortuitously do exactly what the programmer intended.
An integer overflow is a type of an arithmetic overflow error when the result of an integer operation does not fit within the allocated memory space. Instead of an error in the program, it usually causes the result to be unexpected.
You misunderstand the reason for undefined behavior. The reason is not memory corruption around the integer - it will always occupy the same size which integers occupy - but the underlying arithmetics.
Since signed integers are not required to be encoded in 2's complement, there can not be specific guidance on what is going to happen when they overflow. Different encoding or CPU behavior can cause different outcomes of overflow, including, for example, program kills due to traps.
And as with all undefined behavior, even if your hardware uses 2's complement for its arithmetic and has defined rules for overflow, compilers are not bound by them. For example, for a long time GCC optimized away any checks which would only come true in a 2's-complement environment. For instance, if (x > x + 1) f()
is going to be removed from optimized code, as signed overflow is undefined behavior, meaning it never happens (from compiler's view, programs never contain code producing undefined behavior), meaning x
can never be greater than x + 1
.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With