Stack resident buffer overflow on 64-bit?

Question

I'm studying some security related things and right now I'm playing around with my own stack. What I'm doing should be very trivial, I'm not even trying to execute the stack, simply to show that I can get control over the instruction pointer on my 64-bit system. I have turned off all protection mechanisms I'm aware of just to be able to play with it (NX-bit, ASLR, also compiling with -fno-stack-protector -z execstack). I don't have that much experience with 64-bit assembly and after spending some time searching and experimenting myself I'm wondering if anyone could shed some light on an issue I'm experiencing.

I have a program (source code below) which simply copies a string into a stack resident buffer with no bounds checking. However when I overwrite with a series of 0x41 I'm expecting to see the RIP be set to 0x4141414141414141, instead I'm finding that my RBP gets set to this value. I do get a segmentation fault, but RIP does not get updated to this (illegal) value at the execution of the RET instruction, even if RSP is set to a legal value. I have even verified in GDB that there is readlable memory containing a series of 0x41's at RSP immediately prior to the RET instruction.

I was under the impression that the LEAVE instruction did:

MOV (E)SP, (E)BP

POP (E)BP

However on 64-bit, the "LEAVEQ" instruction seems to do (similar to):

MOV RBP, QWORD PTR [RSP]

I'm thinking it does this simply from observing the contents of all registers before and after execution of this instruction. LEAVEQ seems to be just a context dependent name of the RET instruction though (which GDB's disassembler gives it), as it is still just a 0xC9.

And the RET instruction seems to do something with the RBP register, perhaps dereferencing it? I was under the impression that RET did (similar to):

MOV RIP, QWORD PTR [RSP]

However like I mentioned, it seems to dereference RBP, I'm thinking it does this because I get a segmentation fault when no other register seems to contain an illegal value.

Source code for the program:

#include <stdio.h>
#include <string.h>

int vuln_function(int argc,char *argv[])
{
    char buffer[512];

    for(int i = 0; i < 512; i++) {
        buffer[i] = 0x42;
    }

    printf("The buffer is at %p\n",buffer);

    if(argc > 1) {
        strcpy(buffer,argv[1]);
    }

    return 0;
}    

int main(int argc,char *argv[])
{
    vuln_function(argc,argv);

    return 0;
}

The for loop is just there to fill the legal part of the buffer with 0x42, which makes it easy to see in the debugger where it is, before the overflow.

Excerpt of debugging session follows:

(gdb) disas vulnerable
Dump of assembler code for function vulnerable:
   0x000000000040056c <+0>:     push   rbp
   0x000000000040056d <+1>:     mov    rbp,rsp
   0x0000000000400570 <+4>:     sub    rsp,0x220
   0x0000000000400577 <+11>:    mov    DWORD PTR [rbp-0x214],edi
   0x000000000040057d <+17>:    mov    QWORD PTR [rbp-0x220],rsi
   0x0000000000400584 <+24>:    mov    DWORD PTR [rbp-0x4],0x0
   0x000000000040058b <+31>:    jmp    0x40059e <vulnerable+50>
   0x000000000040058d <+33>:    mov    eax,DWORD PTR [rbp-0x4]
   0x0000000000400590 <+36>:    cdqe   
   0x0000000000400592 <+38>:    mov    BYTE PTR [rbp+rax*1-0x210],0x42
   0x000000000040059a <+46>:    add    DWORD PTR [rbp-0x4],0x1
   0x000000000040059e <+50>:    cmp    DWORD PTR [rbp-0x4],0x1ff
   0x00000000004005a5 <+57>:    jle    0x40058d <vulnerable+33>
   0x00000000004005a7 <+59>:    lea    rax,[rbp-0x210]
   0x00000000004005ae <+66>:    mov    rsi,rax
   0x00000000004005b1 <+69>:    mov    edi,0x40070c
   0x00000000004005b6 <+74>:    mov    eax,0x0
   0x00000000004005bb <+79>:    call   0x4003d8 <printf@plt>
   0x00000000004005c0 <+84>:    cmp    DWORD PTR [rbp-0x214],0x1
   0x00000000004005c7 <+91>:    jle    0x4005e9 <vulnerable+125>
   0x00000000004005c9 <+93>:    mov    rax,QWORD PTR [rbp-0x220]
   0x00000000004005d0 <+100>:   add    rax,0x8
   0x00000000004005d4 <+104>:   mov    rdx,QWORD PTR [rax]
   0x00000000004005d7 <+107>:   lea    rax,[rbp-0x210]
   0x00000000004005de <+114>:   mov    rsi,rdx
   0x00000000004005e1 <+117>:   mov    rdi,rax
   0x00000000004005e4 <+120>:   call   0x4003f8 <strcpy@plt>
   0x00000000004005e9 <+125>:   mov    eax,0x0
   0x00000000004005ee <+130>:   leave  
   0x00000000004005ef <+131>:   ret    

I break right before the call to strcpy(), but after the buffer has been filled with 0x42's.

(gdb) break *0x00000000004005e1

The program is executed with 650 0x41's as argument, this should be plenty to overwrite the return address on the stack.

(gdb) run `perl -e 'print "A"x650'`

I search the memory for the return address 0x00400610 (which I found from looking at the disassembly of main).

(gdb) find $rsp, +1024, 0x00400610
0x7fffffffda98
1 pattern found.

I examine the memory with x/200x and get a nice overview which I have omitted here because of its size, but I can clearly see the 0x42 that denote the legal size of the buffer, and the return address.

0x7fffffffda90: 0xffffdab0      0x00007fff      0x00400610      0x00000000

New breakpoint just after strcpy():

(gdb) break *0x00000000004005e9
(gdb) set disassemble-next-line on
(gdb) si
19 }
=> 0x00000000004005ee <vulnerable+130>:  c9     leave  
   0x00000000004005ef <vulnerable+131>:  c3     ret    
(gdb) i r
rax            0x0      0
rbx            0x0      0
rcx            0x4141414141414141       4702111234474983745
rdx            0x414141 4276545
rsi            0x7fffffffe17a   140737488347514
rdi            0x7fffffffdb00   140737488345856
rbp            0x7fffffffda90   0x7fffffffda90
rsp            0x7fffffffd870   0x7fffffffd870
r8             0x1      1
r9             0x270    624
r10            0x6      6
r11            0x7ffff7b9fff0   140737349550064
r12            0x400410 4195344
r13            0x7fffffffdb90   140737488346000
r14            0x0      0
r15            0x0      0
rip            0x4005ee 0x4005ee <vulnerable+130>

   0x00000000004005ee <vulnerable+130>:  c9     leave  
=> 0x00000000004005ef <vulnerable+131>:  c3     ret    
(gdb) i r
rax            0x0      0
rbx            0x0      0
rcx            0x4141414141414141       4702111234474983745
rdx            0x414141 4276545
rsi            0x7fffffffe17a   140737488347514
rdi            0x7fffffffdb00   140737488345856
rbp            0x4141414141414141       0x4141414141414141
rsp            0x7fffffffda98   0x7fffffffda98
r8             0x1      1
r9             0x270    624
r10            0x6      6
r11            0x7ffff7b9fff0   140737349550064
r12            0x400410 4195344
r13            0x7fffffffdb90   140737488346000
r14            0x0      0
r15            0x0      0
rip            0x4005ef 0x4005ef <vulnerable+131>
(gdb) si

Program received signal SIGSEGV, Segmentation fault.
   0x00000000004005ee <vulnerable+130>:  c9     leave  
=> 0x00000000004005ef <vulnerable+131>:  c3     ret    
(gdb) i r
rax            0x0      0
rbx            0x0      0
rcx            0x4141414141414141       4702111234474983745
rdx            0x414141 4276545
rsi            0x7fffffffe17a   140737488347514
rdi            0x7fffffffdb00   140737488345856
rbp            0x4141414141414141       0x4141414141414141
rsp            0x7fffffffda98   0x7fffffffda98
r8             0x1      1
r9             0x270    624
r10            0x6      6
r11            0x7ffff7b9fff0   140737349550064
r12            0x400410 4195344
r13            0x7fffffffdb90   140737488346000
r14            0x0      0
r15            0x0      0
rip            0x4005ef 0x4005ef <vulnerable+131>

I verify that the return address has been overwritten and I should have expected to see RIP get set to this address:

(gdb) x/4x 0x7fffffffda90
0x7fffffffda90: 0x41414141      0x41414141      0x41414141      0x41414141
(gdb) x/4x $rsp          
0x7fffffffda98: 0x41414141      0x41414141      0x41414141      0x41414141

Yet RIP is clearly:

rip            0x4005ef 0x4005ef <vulnerable+131>

Why has not RIP gotten updated as I'm expecting? What does LEAVEQ and RETQ really do on 64-bit? In short, what am I missing here? I have tried to omit the compiler arguments when compiling just to see if it makes any difference, it doesn't seem to make any difference.

Answer 1

Those two instructions are doing exactly what you expect them to do. You have overwritten the previous stack frame with 0x41's so when you hit the leaveq, you are doing this:

mov rsp, rbp
pop rpb

Now rsp points to where rbp did before. However, you have overwritten that region of memory, so when you do the pop rbp, the hardware is essentially doing this

mov rbp, [rsp]
add rsp,1

But [rsp] now has 0x41's. So this is why you're seeing rbp get filled with that value.

As for why rip isn't getting set like you expect, it's because ret is setting the rip to 0x41 and then generating an exception (page fault) on the instruction fetch. I wouldn't rely on GDB to show the right thing in this case. You should try overwriting the return value with a valid address within the program's text segment and you likely won't see this weird behavior.