How to call a function through a function pointer passed as an argument?

Question

How do I call passed function (*f2) in third argument of f1 function in assembly? Declaration looks like that:

extern float f1(int v1, float v2, float (*f2)(int v3, float v4));

I want to pass v1 to v3, v2 to v4 , call function f2, and return value

f1:
    push rbp           
    mov rbp, rsp

    mov rdx, rdi ; v1 to v3
    mov xmm1, xmm0 ; v2 to v4
    call ??? 
    mov xmm0, xmm1

    mov rsp, rbp       
    pop rbp    
ret

what do I put in place of the question marks?

Answer 1

There is no such thing as "Abi64". Since you tagged the question MASM, we can guess that you're using the Windows platform, and clearly the "64" means that this is 64-bit code, so that does narrow down the possibilities tremendously. However, there are still two common calling conventions for 64-bit code on Windows. One of them is __vectorcall and the other is the Microsoft x64 calling convention (the one that was originally invented to make all other calling conventions obsolete but…didn't).

Since the Microsoft x64 calling convention is the most common, and in this particular case, using __vectorcall wouldn't change anything, I'll assume it's the one that you're using. And then the code required becomes absolutely trivial. All you need to do is jump from f1 to f2, since the stack will be set up identically. f1's first two parameters are the two parameters that should be passed to f2, and the return value of f2 is the return value of f1. Therefore:

f1:
    rex_jmp  r8    ; third parameter (pointer to f2) is passed in r8

This is not only trivial to write, but it is the most optimal implementation for both size and speed.
You can even modify the v1 or v2 parameters beforehand if you want, e.g.:

f1:
    inc      ecx        ; increment v1 (passed in ecx)

    ; multiply v2 (xmm1) by v1 (ecx)
    movd     xmm0, ecx
    cvtdq2ps xmm0, xmm0
    mulss    xmm1, xmm0

    rex_jmp  r8    ; third parameter (pointer to f2) is passed in r8

In case you wanted to do something more complicated, here's how it would work:

f1:
    sub   rsp, 40     ; allocate the required space on the stack
    call  r8          ; call f2 through the pointer, passed in r8
    add   rsp, 40     ; clean up the stack
    ret

Note that you do not need the prologue/epilogue code that you've shown in the question, though it won't hurt anything if you choose to include it.

However, the shuffling of parameters that you were doing in the example code shown in the question is wrong! In the Microsoft x64 calling convention, the first up-to-four integer arguments are passed in registers, from left to right, in RCX, RDX, R8, and R9. All other integer arguments are passed on the stack. The first up-to-four floating-point values are also passed in registers, from left to right, in XMM0, XMM1, XMM2, and XMM3. The rest are passed on the stack, along with structs too large for registers.

The weird thing, though, is that the slots are "fixed", so only 4 total register args can be used, even when you have a mix of integer and fp args. Thus:

╔═══════════╦══════════════════════════╗
║           ║           TYPE           ║
║ PARAMETER ╠═════════╦════════════════╣
║           ║ Integer ║ Floating-Point ║
╠═══════════╬═════════╬════════════════╣
║ First     ║   RCX   ║      XMM0      ║
╠═══════════╬═════════╬════════════════╣
║ Second    ║   RDX   ║      XMM1      ║
╠═══════════╬═════════╬════════════════╣
║ Third     ║   R8    ║      XMM2      ║
╠═══════════╬═════════╬════════════════╣
║ Fourth    ║   R9    ║      XMM3      ║
╠═══════════╬═════════╩════════════════╣
║ (rest)    ║         on stack         ║
╚═══════════╩══════════════════════════╝

It doesn't matter that the second parameter is the first floating-point value that is being passed. It doesn't go in XMM0 because it's the first floating-point value, it goes in XMM1 because it's the second parameter and therefore in the second "slot". (This is different from the x86-64 System V ABI, where the first 6 integer args go in registers, whether or not there are FP args).

More detailed documentation on Windows parameter passing is available here, including examples.