Implementing a sandbox with custom stack on Windows 64-bit

Question

I am currently investigating how to implement a sandbox (similar to Google's NaCl project ) where I can run untrusted x86 code (restricted instruction set) in such a way that it cannot harm the rest of my process.

Unlike with NaCl, the untrusted code will not run in a separate process but the same process as the host application. So, one crucial step is to get Windows' structured exception handling right in order to catch errors (like invalid memory access or div by 0) and gracefully terminate the sandbox before Windows kills my host application. (NaCl doesn't face these issues. The sandbox is a separate process and simply gets killed in case of an error.)

Furthermore, the sandboxed code should not use the host applications stack but run on some separate "stack" which is allocated by myself.

Exactly this combination (exception handling in presence of a custom allocated stack) is twisting my mind. I have checked the language implementations of Go and Factor which do similar things and with this help got something running.

But there are still some open questions and uncertainties. So I thought I'll use the fantastic knowledge of Stack Overflow to get some opinions :-)

The following is a working code snippet cut down to the core issues:

code.cpp

#include <Windows.h>
extern "C" void Sandbox();

// just a low level helper to print "msg"
extern "C" void Write(const char* msg)
{
    WriteFile(GetStdHandle(STD_OUTPUT_HANDLE),
              msg, (DWORD)strlen(msg), NULL, NULL);
}

// should be called first on error and continue exception handling
LONG __stdcall GlobalExceptionHandler(_EXCEPTION_POINTERS*)
{
    Write("GEH ");
    return EXCEPTION_CONTINUE_SEARCH;
}

// should be called afterwards on error and terminate the process
// of course this is just a stub to simplify the issue
// in real world it would just terminate the sandbox
extern "C" EXCEPTION_DISPOSITION __stdcall FrameExceptionHandler(
        PEXCEPTION_RECORD, ULONG64, PCONTEXT, PVOID)
{
    Write("FEH ");
    ExitProcess(42);
}

void main()
{
    AddVectoredExceptionHandler(1, GlobalExceptionHandler);
    Sandbox();
    // never reach this...
    ExitProcess(23);
}

code.asm

EXTERN FrameExceptionHandler:PROC
EXTERN malloc:PROC

.code

Handler:
    jmp FrameExceptionHandler

Sandbox PROC FRAME : Handler
    ; function prologue compliant with Windows x86_64 calling conventions
    ; saves rsp to the "frame-pointer" r15
    push r15
    .PUSHREG r15
    sub rsp, 20h
    .ALLOCSTACK(20h)
    mov r15, rsp
    .SETFRAME r15, 0h
    .ENDPROLOG

    ; set rsp to the top of a "heap allocated stack" of size 0x10000 bytes
    mov rcx, 10000h
    call malloc
    lea rsp, [rax+10000h]

    ; got this from implementation of the Go language runtime:
    ; while unwinding the stack, Windows sanity checks the values of
    ; RSP to be within stack-bounds. Of course RSP is set to our
    ; "heap allocated stack" and not within the bounds of what Windows
    ; thinks should be the stack.
    ; Fix this by adjusting StackBase and StackEnd in the TIB (thread
    ; information block), so that basically the stack is unbounded:
    ; StackBase = 0xffffffffffffffff, StackEnd = 0x0000000000000000
    mov rcx, 0FFFFFFFFFFFFFFFFh
    mov gs:[008h], rcx
    mov rcx, 0
    mov gs:[010h], rcx


    ; trigger an access error by reading invalid memory
    mov rax, 0DEADBEEFh
    mov rax, [rax]

    ; function epilogue - will never get here
    mov rax, 0
    add rsp, 28h
    ret
Sandbox ENDP

end

Running this will print "GEH FEH " and then gracefully exit with code 42.

Does anyone have more insight in this set StackBase & StackEnd "hack"? I tried to narrow the stack limits to something like:

    mov gs:[008h], rsp
    mov gs:[010h], rax    ; rax is the address returned by malloc

But it does not work. It prints "GEH " and then crashes due to unhandled exceptions. FrameExceptionHandler() will never be executed.

I also tried more relaxed boundaries that include the "heap allocated stack" as well as the stack allocated by Windows. But it doesn't help.

Another question is, whether you know any other traps I can run into. For example I noticed that Windows does not like it if RSP is uneven (I guess because you can never get to an uneven RSP by doing 2/4/8 byte PUSHes and POPs on a 16-byte aligned stack pointer).

Thanks, Jonas

Answer 1

Running untrusted, 3rd party code in the same process is asking for trouble. That code can kill your process in various ways. E.g. it can call exit() on failures, ask for lots of memory, or write to memory allocated by your threads.

The more secure yet not so hard solution is to run this code in a different process, similar to what Chrome is doing. Every Chrome extension is run in a different process and data is passed between processes.

Your application can launch a separate process and communicate with it though a pipe, windows message, shared memory (memory mapped file) for sharing large data, etc.

A plugin (usually) implements an interface so you'll need to write a proxy object to abstract the fact that the plugin resides in another process and hide the IPC complexity that comes with multi process apps. gSoap is such a framework, others exist.