Is it possible to replace every instance of a particular function with a dummy in a compiled binary?

Question

Is it possible to alter the way that an existing x86-64 binary references and/or calls one particular function. Specifically, is it possible to alter the binary such nothing happens (similar to a nop) at the times when that function would normally have executed?

I realize that there are powerful speciality tools out there (ie decompilers/disassemblers) for just this sort of task, but what I'm really wondering is if the executable formats are human-readable "enough" to be able to do this sort of thing (on small programs, at least) with just vim and a hex editor.

Are certain executable file formats (eg mach-o, elf, whatever the heck windows uses, etc.) more readable than others? Are they all just completely incomprehensible gibberish? Any expert views and/or good jumping off points/references would be greatly appreciated.

Disclaimer

Someone came by and quickly downvoted the initial version of this question, so I want to make this perfectly clear: I am not interested in disabling any serial or security checks or anything of the sort. Originally I had wanted a program to stop making a really irritating noise, but now I'm just curious about how compilers and executables work.

I'm in this for the educational value, and I think that other people on SE will be interested in the answer. However, I appreciate that others might not be as comfortable with this topic. If you have a concern about something I've said, please leave a comment and I promise I'll change my post.

Answer 1

This is trivial to do when the function in question is in the binary itself and uses standard calling conventions. Example:

void make_noise() { printf("Quack!\n"); }
int fn1() { puts("fn1"); make_noise(); return 1; }
int fn2() { puts("fn2"); make_noise(); return 2; }
int main() { puts("main"); return fn1() + fn2() - 3; }

gcc -w t.c -o a.out && ./a.out

This outputs (expected):

main
fn1
Quack!
fn2
Quack!

Now let's get rid of the noise:

gdb -q --write ./a.out
(gdb) disas/r make_noise
Dump of assembler code for function make_noise:
  0x000000000040052d <+0>:     55      push   %rbp
  0x000000000040052e <+1>:     48 89 e5        mov    %rsp,%rbp
  0x0000000000400531 <+4>:     bf 34 06 40 00  mov    $0x400634,%edi
  0x0000000000400536 <+9>:     e8 d5 fe ff ff  callq  0x400410 <puts@plt>
  0x000000000040053b <+14>:    5d      pop    %rbp
  0x000000000040053c <+15>:    c3      retq   
End of assembler dump. 

This tells us a few things:

1. The function that we want to get rid of starts at address 0x40052d
2. The op-code of retq instruction is 0xC3.

Let's patch retq as the first instruction of make_noise, and see what happens:

(gdb) set *(char*)0x40052d = 0xc3
(gdb) disas make_noise
Dump of assembler code for function make_noise:
  0x000000000040052d <+0>:     retq   
  0x000000000040052e <+1>:     mov    %rsp,%rbp
  0x0000000000400531 <+4>:     mov    $0x400634,%edi
  0x0000000000400536 <+9>:     callq  0x400410 <puts@plt>
  0x000000000040053b <+14>:    pop    %rbp
  0x000000000040053c <+15>:    retq   
End of assembler dump.

It worked!

(gdb) q
Segmentation fault (core dumped)   ## This is a long-standing GDB bug

And now let's run patched binary:

$ ./a.out
main
fn1
fn2

Voila! No noise.

If the function is in a different binary, LD_PRELOAD techniques mentioned by Florian Weimer is usually easier than binary patching.

Answer 2

ELF dynamic linking implementations often support LD_PRELOAD and LD_AUDIT modules, which can both intercept calls into another shared object. LD_AUDIT offers more control, and exists on GNU/Linux (but the Solaris documentation is the canonical reference).

For calls within the same shared object, this may not be possible if the target function is not exported (or the call is executed via a hidden alias; glibc does this a lot). If you have debugging information, you can use systemtap to intercept the call. If the function is inlined, intercepting the call might not be possible even with systemtap because there is no exact place in the instruction stream where the call takes place.