Convert inline assembly code to C++

Question

I am working on a cpp project. The project need to be migrated to 64 bit. It contains some Inline assembly code which cannot compile on x64. This is the Function which contain the assembly code:

void ExternalFunctionCall::callFunction(ArgType resultType, void* resultBuffer)
{
#if defined(_NT_) || defined(__OS2__)

    // I386

    // just copy the args buffer to the stack (it's already layed out correctly)
    int* begin = m_argsBegin;
    int* ptr = m_argsEnd;
    int arr[1000], i=0;
    while (ptr > begin) {
        int val = *(--ptr);

        __asm push val
    }

    void* functionAddress = m_functionAddress;

    // call the function & handle the return value.  use __stdcall calling convention
    switch (resultType) {
    case voidType:
        __asm {
            call functionAddress
        }
        break;
    case pointerType:
    case int32Type:
        __asm {
            call functionAddress
            mov ebx, resultBuffer
            mov dword ptr [ebx],eax
        }
        break;
    case floatType:
        __asm {
            call functionAddress
            mov ebx, resultBuffer
            fstp dword ptr [ebx]
        }
        break;
    case doubleType:
        __asm {
            call functionAddress
            mov ebx, resultBuffer
            fstp qword ptr [ebx]
        }
        break;
    }

I used stack, array to migrate this "asm push val" but didn't work. Although, It don't throw any compilation error but logic didn't worked.

So, I want to ask, What can I use in C++ instead of "__asm push val". Any help will be appreciated.

Answer 1

The problem isn't solvable in general; that's because the comment,

// call the function & handle the return value.  use __stdcall calling convention

indicates reliance on 32bit calling conventions.

In 32bit x86, stdcall means that all arguments are passed on the stack, in reverse order (i.e. the last arg is pushed first. That is, if arg[0] is at addr then arg[1], no matter it's type, is at addr + sizeof(arg[0])). That's the reason why the following code in your sample:

// just copy the args buffer to the stack (it's already layed out correctly)
int* begin = m_argsBegin;
int* ptr = m_argsEnd;
int arr[1000], i=0;
while (ptr > begin) {
    int val = *(--ptr);

    __asm push val
}

actually can work - because it simply doesn't matter what exactly is there, what type the arguments are; all that's relevant is that each and every one of them is in a known memory location, and known-to-be-consecutive-in-memory. If you know that argument N is at addr, then you can know argument N+1 is at addr + sizeof(arg[N]).

That's what the comment says, "it's already layed out correctly" - unfortunately, this is not true in 64bit mode. The code therefore cannot be "ported"; there is no equivalent to port to.

Calling conventions that are at least partially register based - as is Win64 on x64 (64bit x86) behave differently. For those, it depends on what types of arguments the called functions take (you can, in Windows, pass four integer-type args in general-purpose registers plus some float-type args in XMM registers). You therefore need to know more about the signature (prototype) of the function you call than simply "it takes N arguments" in order to be able to correctly marshal args from an "anycall" type wrapper like the above. In 64bit mode, for every function you'd wish to call through your wrapper, you need to know not just how many args there are in total, but also how many are in general-purpose regs, how many in XMM regs, and how many on the stack.

The "call a func through a pointer and copy the return value to a known location" part is portable and can be expressed in plain C/C++. But the part that gets the arguments for this, as per above, does not port in any straightforward way between 32bit stdcall and any 64bit x86 calling convention I know of (neither Win64/x64 nor the UN*X x86_64 conventions allow to predict both the locations and total stack memory usage of all arguments for a function given just the number and types of the args but not their ordering).

What exactly you need to do depends much more on the callers/users of the above class ExternalFunctionCall than on the small little sample of inline assembly you've shown. It's particularly necessary to know how the members m_argsBegin and m_argsEnd are initialized and where. Can you provide a few more details on how the class looks like (all member variables/functions), as well as examples on its actual use ?