"xor eax, ebp" being used in C++ compiler output

Question

These are buffer overrun protection methods, and have nothing to do with compiler optimisation. MSVC will (if you specify the /GS switch) push a security cookie onto the stack near the return address so that it can detect a common case of stack corruption.

Stack corruption can either be caused by bad code along the lines of:

char buff[5];
strcpy (buff, "Man, this string is waaay too long!!");

or by malicious users taking advantage of bad coding practices, like the use of scanf ("%s", myBuff) for user input. Carefully crafted attacks like that can suborn your program to do things you probably don't want it to.

By placing a cookie close to the return address, a large number of bugs (and attack vectors) can be prevented, simply due to the fact that the memory corruptions tend to be sequential in nature. In other words, if you've overwritten the return address, it's probably because you started writing on one side of the cookie and corrupted memory all the way up to the return address on the other side of the cookie (hence the cookie will be overwritten as well).

It doesn't catch all bugs since you may have some code like:

char buff[5];
buff[87] = 'x';

which could potentially corrupt the return address without touching the cookie. But it will catch all those malicious ones which rely on entering a longer string than expected, which corrupt up to the return address (including cookie).

The sequence you're probably seeing in the code is something like:

mov  eax, dword ptr ds:___sec_cookie   ; fixed value.
xor  eax, ebp                          ; adjust based on base pointer.
mov  [ebp+SOMETHING], eax              ; store adjusted value.

which is customising the cookie, depending on the current base pointer.

This will change what is actually put on the stack at each stack level (and also depending on parameter count and sizes as well) and is probably an attempt to further secure the code from malicious intent, by ensuring a variable signature is written to the stack rather than a fixed value (otherwise the attacker could enter characters including a valid cookie).

And the sequence at the end will run something like this:

mov  ecx, [ebp+SOMETHING]              ; get the adjusted cookie.
xor  ecx, ebp                          ; un-adjust it, since
                                       ;   ((N xor X) xor X) == N.
call @__sec_check_cookie               ; check the cookie.

It's basically just the reverse process of that described above. The @__sec_check_cookie call will only return if ecx is set to the correct cookie value. Otherwise it will raise a fault, as confirmed here:

The __security_check_cookie() routine is straightforward: if the cookie was unchanged, it executes the RET instruction and ends the function call. If the cookie fails to match, the routine calls report_failure().

The report_failure() function then calls __security_error_handler(). Both functions are defined in the seccook.c file of the C run-time (CRT) source files.

CRT support is needed to make these security checks work. When a security check failure occurs, control of the program is passed to __security_error_handler(), which is summarized here:

void __cdecl __security_error_handler(int code, void *data)
{
    if (user_handler != NULL) {
      __try {
        user_handler(code, data);
      } __except (EXCEPTION_EXECUTE_HANDLER) {}
    } else {
      //...prepare outmsg...

      __crtMessageBoxA(
          outmsg,
          "Microsoft Visual C++ Runtime Library",
          MB_OK|MB_ICONHAND|MB_SETFOREGROUND|MB_TASKMODAL);
    }
    _exit(3);
}

By default, an application that fails a security check displays a dialog that states "Buffer overrun detected!". When the dialog is dismissed, the application terminates.