-finstrument-functions doesn't work with dynamically loaded g++ shared objects (.so)

Question

I am testing -finstrument-functions with g++ shared object (.so) files on Ubuntu these days. I found a strange behavior that -finstrument-functions seems to work only if the library is statically linked. If I link to the library with dlopen/dlsym etc., the functionality of the code still works, but it won't call the __cyg_profile* functions.

Here are some codes to quickly reproduce the problem:

MyLib.h

#ifndef __MYLIB_H__
#define __MYLIB_H__
class MyLib
{
public:
    void sayHello();
};
#endif

MyLib.cpp

#include "MyLib.h"
#include <iostream>
using namespace std;

void MyLib::sayHello()
{
    cout<<"Hello"<<endl;
}

MyLibStub.cpp (C interface to the .so)

#include "MyLib.h"

extern "C" void LoadMyLib ()
{
    MyLib().sayHello();
}

Trace.cpp

#include <stdio.h>
#ifdef __cplusplus
extern "C"
{
    void __cyg_profile_func_enter(void *this_fn, void *call_site)
        __attribute__((no_instrument_function));
    void __cyg_profile_func_exit(void *this_fn, void *call_site)
        __attribute__((no_instrument_function));
}
#endif

void __cyg_profile_func_enter(void* this_fn, void* call_site)
{
    printf("entering %p\n", (int*)this_fn);
}

void __cyg_profile_func_exit(void* this_fn, void* call_site)
{
    printf("exiting %p\n", (int*)this_fn);
}

MainStatic.cpp

#include <iostream>
using namespace std;

extern "C" void LoadMyLib ();

int main()
{
    LoadMyLib();
    return 0;
}

MainDynamic.cpp

#include <iostream>
#include <dlfcn.h>

const char* pszLibName  = "libMyLib.so.0.0";
const char* pszFuncName  = "LoadMyLib";

int main()
{
    void* pLibHandle = dlopen(pszLibName, RTLD_NOW);
    if(!pLibHandle) {
        return 1;
    }
    void (*pFuncLoad)() = 0;
    //Resolve the function in MyLibStub.cpp
    pFuncLoad = (void (*)())dlsym(pLibHandle, pszFuncName);
    if(!pFuncLoad) {
        return 1;
    }
    pFuncLoad();
    dlclose(pLibHandle);
    return 0;
}

and compile with the following commands (under Ubuntu 11.10):

g++ -g -finstrument-functions -Wall -Wl,-soname,libMyLib.so.0 -shared -fPIC -rdynamic MyLib.cpp MyLibStub.cpp Trace.cpp -o libMyLib.so.0.0  
ln -s libMyLib.so.0.0 libMyLib.so.0  
ln -s libMyLib.so.0.0 libMyLib.so  
g++ MainStatic.cpp -g -Wall -lMyLib -L./ -o MainStatic   
g++ MainDynamic.cpp -g -Wall -ldl -o MainDynamic

when called with ./MainStatic

it gives something like:

entering 0xb777693f
entering 0xb777689b
exiting 0xb777689b
exiting 0xb777693f
entering 0xb7776998
entering 0xb777680c
Hello
exiting 0xb777680c
exiting 0xb7776998

however, when called with ./MainDynamic

it only gives a "Hello".

Hello

Does anybody here know why there is such difference between statically and dynamically linked libraries? Is there any solution to make it work even when dynamically loaded? Thanks in advance.

Answer 1

This behavior is expected.

In order to understand it, you first need to know that the dynamic loader searches for symbols using a linked list, in the order that different ELF images were loaded. At the head of that list is the main executable itself, followed by all libraries directly linked to it. When you dlopen() some library, it gets appended to the tail of the list.

So when the code in the library you just loaded calls __cyg_profile_func_enter, the loader searches the list for the first definition of that function. That first definition happens to be the default one, provided by libc.so.6, which is near the end of the list, but is before your dlopen()ed library.

You can observe all of this by running:

LD_DEBUG=symbols,bindings ./MainDynamic

and looking for __cyg_profile_func_enter in the output.

So, what do you have to do in order to see your instrumentation? You have to get your own __cyg_profile_func_enter somewhere before the one from libc.so.6. One way to do that is to link it into your main executable. Or link it into a shared library that is directly linked to your executable (i.e. not dlopen()d one).

Once you do that, your implementation will be the first one on the list, and it will win over the one in libc.so.6, and you will see the output it generates.