Running 32-bit code in 64-bit process on Linux - memory access

Question

I'm experimenting with running 32-bit code inside a 64-bit Linux process. The 32-bit code is completely self-contained, it makes direct IA32 system calls on its own. If I were to load this code in a 32-bit process, it would run just fine.

Initially, I thought I could just allocate a stack for the 32-bit code, switch to it and everything would work fine, but that didn't go so well. Mainly because stack-related instructions (POP/PUSH/...) were doing 8-byte shifts instead of 4 bytes.

By googling, I learned that I can transition to 32-bit mode by switching to segment selector 0x23. Unfortunately, segments are something I know very little about.

I am able to transition to 32-bit mode with something like this (inline AT&T assembly):

movl $0x23, 4(%%rsp) // segment selector 0x23
movq %0, %%rax
movl %%eax, (%%rsp) // target 32-bit address to jump to
lret

Where %0 contains a 32-bit address of where the code is mapped. The code starts running, I can see that PUSH/POP now works the way it should, but it crashes even earlier than when I ran the code in 64-bit mode on a seemingly innocuous instruction:

0x8fe48201      mov    0xa483c(%rbx),%ecx

Where %rbx (or more like %ebx since this code is already 32-bit, GDB just doesn't know that) contains 0x8fe48200. The address it's trying to read from (0x8feeca3c) is valid and readable (according to /proc/XXX/maps) and when I read it from within GDB, it contains the expect value.

Yet, Linux sends a SIGSEGV to the process on this instruction and the faulting address is 0 (as reported by strace or p $_siginfo._sifields._sigfault.si_addr inside gdb). Somehow it seems 0x8feeca3c is not a valid address in the 32-bit realm.

Any ideas how to proceed?

UPDATE: I have written a minimal example that segfaults reading address 0, although address 0 is not really being referenced. It seems that reading any addresses in memory fails (even reading the address of the instruction that has just been executed!), although stack operations work OK.

#include <sys/mman.h>
#include <string.h>
#include <stdio.h>
#include <stdint.h>

// 32-bit code we're executing
const unsigned char instructions[] = {
        0x6a, 0, // push 0
        0x58, // popl %eax
        0xe8, 0, 0, 0, 0, // call the next line to get our location in memory
        0x5b, // pop %ebx
        // THE FOLLOWING mov SEGFAULTS, but it is well within the mapped area (which has size 0x3000)
        // A simpler "mov (%ebx), %eax" (0x8b, 0x03) would fail as well
        0x8b, 0x83, 0, 0x20, 0, 0, // mov 0x2000(%ebx), %eax
        0xf4 // hlt, not reached
};

int main()
{
        void* area;
        void* stack;

        area = mmap(NULL, 3*4096, PROT_WRITE|PROT_READ|PROT_EXEC, MAP_ANONYMOUS | MAP_PRIVATE | MAP_32BIT, -1, 0);
        memcpy(area, instructions, sizeof(instructions));

        stack = mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE | MAP_32BIT, -1, 0);
        stack = (void*) (((uint64_t) stack) + 4096 - 4);

        memset(((char*)area) + 2*4096, 0xab, 100); // Place 0xAB in the area we mov from in 32-bit instructions

        // Switch to 32-bit mode and jump into the code
        __asm__ volatile("movq %1, %%rsp;" \
                         "subq $8, %%rsp;" \
                         "movl $0x23, 4(%%rsp);" \
                         "movq %0, %%rax;" \
                         "movl %%eax, (%%rsp);" \
                         "lret" :: "m"(area), "r"(stack) :);
}

Answer 1

Nice question :)

The problem is that ds is still set to zero, in 64 bit mode it's not used. So, you need to reload that and it will work. Changing your initial test push/pop to push $0x2b; pop %ds will do the trick:

const unsigned char instructions[] = {
        0x6a, 0x2b, // push $0x2b
        0x1f, // pop %ds

I extracted the 0x2b value from a 32 bit program. I just kept wondering why push worked. On closer look, ss is set in 64 bit mode too, so it may be safer to copy that to ds as well as to es.