try/catch with __debugbreak()

Question

I am working with a 3rd party C++ DLL that is running __debugbreak() in some scenario, and is not checking IsDebuggerPresent() before doing so. This results in my application "crashing" when that scenario happens outside of a debugger (e.g. end user running the application). I would like to catch this and deal with it myself, or at least ignore it.

I actually have had an unhandled exception filter in place to translate SEH to C++ exceptions for a while, so it's a little strange that it's not working.

::SetUnhandledExceptionFilter(OnUnhandledException);

I've been doing some direct testing, and the standard __try/__except works, so I could wrap every call into the DLL with this as a fallback, but seems to be that if __try/__except works, then ::SetUnhandledExceptionFilter() should also work.

    __try
    {
        __debugbreak();
    }
    __except (EXCEPTION_EXECUTE_HANDLER)
    {
        printf("caught");
    }

try/catch(...) does not work.

    try
    {
        __debugbreak();
    }
    catch (...)
    {
        printf("caught");
    }

_set_se_translator() isn't working either.

From the MSDN documentation at https://msdn.microsoft.com/en-us/library/ms679297(VS.85).aspx it states that it should function as a structured exception. I realize that is the documentation for DebugBreak() but I have tested with that as well and have the same problem, even with "catch(...)".

I am compiling with /EHa.

How can I catch the __debugbreak (asm INT 3), or at least change the behavior?

Answer 1

Breakpoints generate the EXCEPTION_BREAKPOINT structured exception. You cannot use try/catch to catch it because it doesn't get translated to a C++ exception, irrespective of the /EHa switch or _set_se_translator. EXCEPTION_BREAKPOINT is a special exception.

First, you should know that catch blocks and __except blocks execute only after unwinding the stack. This means that execution continues after the handler block, NOT after the call to __debugbreak(). So if you just want to skip EXCEPTION_BREAKPOINT while at the same time continue execution after the int 3 instruction. You should use a vectored exception handler. Here is an example:

// VEH is supported only on Windows XP+ and Windows Server 2003+
#define _WIN32_WINNT 0x05020000

#include <windows.h>
#include <stdio.h>

//AddVectoredExceptionHandler constants:
//CALL_FIRST means call this exception handler first;
//CALL_LAST means call this exception handler last
#define CALL_FIRST 1  
#define CALL_LAST 0

LONG WINAPI
VectoredHandlerBreakPoint(
struct _EXCEPTION_POINTERS *ExceptionInfo
    )
{
    if (ExceptionInfo->ExceptionRecord->ExceptionCode == EXCEPTION_BREAKPOINT)
    {
        /*

        If a debugger is attached, this will never be executed.

        */

        printf("BreakPoint at 0x%x skipped.\n", ExceptionInfo->ExceptionRecord->ExceptionAddress);

        PCONTEXT Context = ExceptionInfo->ContextRecord;

        // The breakpoint instruction is 0xCC (int 3), just one byte in size.
        // Advance to the next instruction. Otherwise, this handler will just be called ad infinitum.
#ifdef _AMD64_
        Context->Rip++;
#else
        Context->Eip++;
#endif    
        // Continue execution from the instruction at Context->Rip/Eip.
        return EXCEPTION_CONTINUE_EXECUTION;
    }

    // IT's not a break intruction. Continue searching for an exception handler.
    return EXCEPTION_CONTINUE_SEARCH;
}

void main()
{
    // Register the vectored exception handler once.
    PVOID hVeh = AddVectoredExceptionHandler(CALL_FIRST, VectoredHandlerBreakPoint);

    if (!hVeh)
    {
        // AddVectoredExceptionHandler failed.
        // Practically, this never happens.
    }

    DebugBreak();

    // Unregister the handler.
    if (hVeh)
        RemoveVectoredExceptionHandler(hVeh);
}

In this way, the breakpoint instruction int 3 will just be skipped and the next instruction will be executed. Also if a debugger is attached, it will handle EXCEPTION_BREAKPOINT for you.

However, if you really want to unwind the stack, you have to use __except(GetExceptionCode() == EXCEPTION_BREAKPOINT ? EXCEPTION_EXECUTE_HANDLER : EXCEPTION_CONTINUE_SEARCH).