Windows native callback only returns 32 bit result on 64 bit platform

Question

I'm calling a DLL function, that takes a callback function, where the callback function returns a pointer to some data structure, and the DLL will perform some action on the data structure.

package main

import (
    "golang.org/x/sys/windows"
    "unsafe"
)

type A struct {}
var a = &A{}

func Callback() uintptr {
    ptr := uintptr(unsafe.Pointer(a))
    fmt.Printf("Address: 0x%X\n", ptr)
    return ptr
}

func main() {
    dll := windows.MustLoadDLL("some.dll")
    init := dll.MustFindProc("init")
    init.Call(windows.NewCallback(Callback))
    fmt.Printf("This never get prints\n")
}

I'm running go on Windows 64-bit machine, the DLL is also 64-bit.

$ go version
go version go1.11.4 windows/amd64

The above code will print the address of the pointer, which is 0xC000080018, but the call to the DLL will fail silently. I don't have the source code to the DLL so I can't make it print the value it gets. So I used IDA pro to dynamically debug the program, and I found that the Go callback function is actually returning 32-bit value instead of 64-bit value. This is the final code of the Go callback routine before it returns back to the DLL:

main.exe:00000000004537E7     mov     eax, [rcx+rdx-8]
main.exe:00000000004537EB     pop     qword ptr [rcx+rdx-8]
main.exe:00000000004537EF     retn

As you can see Go is writing the return value in eax instead of rax, which will discard the higher address, therefore the DLL will only get 0x80018 and caused errors.

Although I know what's wrong with it, I can't figure out how to fix it.

Update:

I've done some digging into Go's implementation of the callback, and found the assembly code at src/runtime/sys_windows_amd64.s. I noticed that in runtime·callbackasm1, the return value is set using MOVL instead of MOVQ. So I changed it to MOVQ and everything worked.

I'm not sure if it's a bug or actually intended, so I will open an issue on GitHub.

Answer 1

This bug was posted as an issue on golang repo (issue 29331) and fixed by the golang team.