Is it possible to change calling convention in gcc for x64?

Question

I’m writing a C program where I intercept the arguments to a function called by pointer before changing them and forwarding the call to the actual function, and this forwarding is done in assembly. I know that when compiling to x86 it’s possible to mark the calling convention on a per-function basis using attribute((stdcall)) so I need to know two things:

1. Is it possible to change the calling convention gcc uses in x64 mode, or is that only available in 32 bit programs?
2. If it is possible, can those, or similar annotations be applied to function pointers so I can ensure the functions are called properly?

I am open to using other compilers if they support this and gcc does not.

Answer 1

GCC supports two calling conventions on x86-64: the System V ABI conventions used on most Unix-like systems, and the Microsoft x64 convention used on Windows. You can choose a specific one for a function with __attribute__((ms_abi)) or __attribute__((sysv_abi)). These attributes can also be applied to function pointers. See https://gcc.gnu.org/onlinedocs/gcc-11.2.0/gcc/x86-Function-Attributes.html#x86-Function-Attributes

Try on godbolt. Notice that foo1 and ptr1 have their arguments passed in ecx and edx per the MS convention, while foo2 and ptr2 use edi and esi as per SysV.

clang behaves the same.

Answer 2

x86-32 (x86)

• stdcall and cdecl exist

x86-64 (x64)

• stdcall and cdecl do not exist
• if used on Windows they both refer to Microsoft x64 calling convention (ms_abi)
• if used on UNIX-like, they both refer to System V AMD64 ABI (sysv_abi)
• ms_abi and sysv_abi exist (for Windows and UNIX-like respectively)

See on Wikipedia