"Trapping" a processes' own sysenter calls in userspace on Windows

Question

I'm working on a runtime non-native binary translator in Windows, and so far I've been able to "trap" interrupts (i.e. INT 0x99) for the OS binaries I'm trying to emulate by using an ugly hack that uses Windows SEH to handle invalid interrupts; but only because the system call vector is different than the one in Windows, allowing me to catch these "soft" exceptions by doing something like this:

static int __stdcall handler_cb(EXCEPTION_POINTERS* pes, ...)
{

    if (pes->ExceptionRecord->ExceptionCode != EXCEPTION_ACCESS_VIOLATION)
        return EXCEPTION_CONTINUE_SEARCH;

    char* instruct = (char*) pes->ContextRecord->Eip;

    if (!instruct)
        handle_invalid_instruction(instruct);   

    switch (instruct[0])
    {
        case 0xcd: // INT
        {
            if (instruct[1] != 0x99) // INT 0x99
                handle_invalid_instruction(instruct);
            handle_syscall_translation();
            ...
        }
        ...
        default:
            halt_and_catch_fire();
    }
    return EXCEPTION_SUCCESS;
}

Which works fairly well (but slowly), the problem with this is that Windows first attempts to handle the instruction/interrupt, and for non-native binaries that use sysenter/sysexit instead of int 0x99, some systenter instructions in the non-native binary are actually valid NT kernel calls themselves when executed, meaning my handler is never called, and worse; the state of the "host" OS is also compromised. Is there any way to "trap" sysenter instructions in Windows? How would I go about doing this?

Answer 1

As far as I know, there is no way (from a user-mode process) to "disable" SYSENTER, so that executing it will generate an exception. (I'm assuming your programs don't try to SYSEXIT, because only Ring 0 can do that).

The only I option I think you have is to do like VirtualBox does, and scan for invalid instructions, replacing them with illegal opcodes or something similar, that you can trap on, and emulate. See 10.4. Details about software virtualization.

To fix these performance and security issues, VirtualBox contains a Code Scanning and Analysis Manager (CSAM), which disassembles guest code, and the Patch Manager (PATM), which can replace it at runtime.

Before executing ring 0 code, CSAM scans it recursively to discover problematic instructions. PATM then performs in-situ patching, i.e. it replaces the instruction with a jump to hypervisor memory where an integrated code generator has placed a more suitable implementation. In reality, this is a very complex task as there are lots of odd situations to be discovered and handled correctly. So, with its current complexity, one could argue that PATM is an advanced in-situ recompiler.