override libc open() library function

Question

I have same overridden open() which is provided by glibc in my library & I have set LD_PRELOAD with my library first, so when the process calls open(), the open which is defined in my library gets called.

THE PROBLEM:- There are several other functions within glibc which calls open() once such example is getpt(), when getpt() calls open(), the open() which is defined in glibc gets called, how would I make getpt() to invoke the open() which is defined in my library().

Constraints: - I dont have the option of compiling glibc.

Answer 1

As correctly stated by tmcguire, the call from posix_openpt to __open is a call to internal symbol, and can not be interposed.

Effectively, glibc developers consider this call an implementation detail, that you have no business of changing.

I am looking at compile time solution

You can't have it.

than run time solution cause run time solution will have performance impact.

Runtime solution need not have any performance impact (besides the overhead of calling your open instead of glibcs).

I only know of one way for a library to interpose glibc internal calls: runtime patching. The idea is to

• find the address of libc.so.6 open (which is an alias for __open),
• locate the boundaries of glibc .text section at runtime
• scan it for CALL __open instructions
• for any such instruction
  • mprotect the page it's on to be writable
  • compute a new instruction that is CALL my_open and patch it "on top" of the original instruction
  • mprotect the page back to read and execute

This is ugly, but it works fine for i*86 (32-bit) Linux, where a CALL can "reach" any other instruction within the 4GB address space. It doesn't work for x86_64, where a CALL is still limited to +/- 2GB, but the distance from your library to glibc could be more than that.

In that case, you need to find a suitable trampoline within libc.so.6 to which you can redirect the original CALL, and into which you could place a register-indirect JMP to your final destination. Fortunately, libc.so.6 usually has multiple suitably-sized unused NOP regions due to function alignment.

Answer 2

I was able to solve it at compile time simply by defining the getpt() function within my library.

This solution is incomplete cause there could be other functions within glibc [other than getpt()] which could call open, then open call within glibc will be called.

I can live with this solution for now, but I would need to fix it completely in future.