Calling convention for dynamically created function in (Visual) C++

Question

I use the following types to create a new function at runtime:

typedef int (*pfunc)(int);

union funcptr {
  pfunc x;
  byte* y;
};

This enables me to write instructions in y and afterwards call the function like this:

byte* p = (byte*)VirtualAllocEx(GetCurrentProcess(), 0, 1<<16, MEM_COMMIT, PAGE_EXECUTE_READWRITE );

// Write some instructions to p

funcptr func;
func.y = p;

int ret = func.x(arg1); // Call the generated function

It is crucial to know how C++ prepare arguments (call convention) and therefore I have looked up the project properties (Visual C++) and I can see it uses __cdecl. It should put arguments on the stack according to: http://msdn.microsoft.com/en-us/library/aa271989(v=vs.60).aspx and http://en.wikipedia.org/wiki/X86_calling_conventions#cdecl but when I look at the assembly generated, the argument is moved to the EAX register.

I want to be absolutely certain how the arguments is prepared. So have I overlooked something about cdecl or is Visual C++ optimizing the call, and if so, how do I ensure it doesn't happen?

Best regards, Lasse Espeholt

Answer 1

The EAX register is used for the return value of the function. You state in the comments that you are compiling using /Gd and so the function will use __cdecl. All the same it would make sense in my view to mark the declaration of your function type pfunc with an explicit __cdecl so that there can be no scope for confusion and mis-match.

Of course, there's nothing to stop you using one of the other calling conventions supported by your compiler. The most important point is that whatever calling convention you settle on, you should explicitly specify the calling convention for the function pointer since the compiler is only responsible for one half of the interface.

Answer 2

At least on Linux, you probably want to use libffi (foreign function interface), and it has even been ported to other systems (including Windows).

And if you want to do machine code generation at runtime, consider using GNU lightning, DotGnu's libjit, LLVM. LuaJit's dynasm etc You could also generate C code into foo.c, get it compiled by forking a gcc -fPIC -shared foo.c -o foo.so command, and dlopen("./foo.so", RTLD_GLOBAL) (and Windows have equivalent ability)