How to execute x86 commands from data buffer?

Question

My question is dedicated mostly to profs and is about using C++ in "strange" way. In C++ there isn't really big difference between pointers to variables and pointers to functions. We can do something useless like this:

char* buff     = new char[32];
void (*func)() = (void (*)())buff;

But we allmost created a function that never existed, right? What if we go further and fill buff with x86 commands stord in a file? OS will never know that a function was created.

#include <iostream>
using namespace std;

// no stack push'ing or pop'ing, nothing to return
void func(void){cout << "Hello?";}

int main()
{
  char* x86_code = new char[6];

  x86_code[0]                 = 0x9A;          // call (far)
  *((__int32*)(x86_code + 1)) = (__int32)func; // load 32-bit address
  x86_code[5]                 = 0xC3;          // ret

  void (*x86_func)(void) = (void (*)(void))x86_code;
  x86_func();

  return 0;
}

Calling x86_func() makes a runtime error (violation reading location 0xFFFFFFFF). How does OS loads it's binaries or modules in RAM if not in this manner? Many thanks.

Answer 1

Indeed you can fill an array with x86 machine code and attempt to execute it. It's called shellcode and managing to make an application or library execute such code when it wasn't intended to is called an "exploit".

Unfortunately it's not that easy, since modern hardware and OSes normally prevent you from executing code from areas that are writable, such as non-const char arrays. This is called W^X (write or execute permissions, but not both) But one can request POSIX-compliant OSes to disable such protection with the mprotect() function. Here's an example that works, because it enables read, write and execute permissions on the array of machine code bytes concerned:

#include <stdio.h>
#include <stdint.h>
#include <sys/mman.h>


typedef int(*FUNKY_POINTER)(void);


char shellcode[] = {
    0xb8, 0x2a, 0x00, 0x00, 0x00, //mov    $0x2a,%eax
    0xc3                          //retq
};



int main(void){
    uintptr_t pageSize        = 4096;
    uintptr_t shellcodeAddr   = (uintptr_t)shellcode;
    uintptr_t pageAlignedAddr = shellcodeAddr & ~(pageSize-1);
    FUNKY_POINTER shellcodeFn = (FUNKY_POINTER)shellcode;

    /* Magic */
    mprotect((void*)pageAlignedAddr,
             (shellcodeAddr - pageAlignedAddr) + sizeof(shellcode),
             PROT_EXEC|PROT_WRITE|PROT_READ);

    printf("The answer to the ultimate question of life, "
           "the universe and everything is %d\n",
           shellcodeFn());

    return 0;
}