Member function call crashes in VS2017

Question

I've been investigating a weird crash after moving a class definition to a different module, and came to a conclusion that the compiler gets confused on how pointers to member functions are defined.

I can't include the whole code, because it is a massive program and I couldn't reproduce it on a smaller example.

Edit: I managed to reproduce the crash on a small example, so I'm editing the whole question to include the new code and assembly.

StatesManager.h:

#pragma once

class StatesManager
{
public:
    bool action();
};

Toolbar.h:

#pragma once

class StatesManager;

class Toolbar
{
public:
    Toolbar( StatesManager* statesManager );

    void action( bool ( StatesManager::*action )( ), bool ( StatesManager::*enabled )( ) const = nullptr );

private:
    StatesManager* statesManager_;
};

Toolbar.cpp:

#include "Toolbar.h"
#include "StatesManager.h"

Toolbar::Toolbar( StatesManager* statesManager ) :
    statesManager_( statesManager )
{
}

void Toolbar::action( bool ( StatesManager::*action )( ), bool ( StatesManager::*enabled )( ) const )
{
    ( statesManager_->*action )( );
}

main.cpp:

#include "StatesManager.h"
#include "toolbar.h"

bool StatesManager::action()
{
    return true;
}

int main()
{
    StatesManager manager;
    Toolbar toolbar( &manager );
    toolbar.action( &StatesManager::action );
    return 0;
}

When this code is called (from another module), I get this assembly:

    ( statesManager_->*action )( );
00007FF743771860  mov         rax,qword ptr [&action]  
00007FF743771867  movsxd      rax,dword ptr [rax+8]  
00007FF74377186B  mov         rcx,qword ptr [this]  
00007FF743771872  add         rax,qword ptr [rcx]  
00007FF743771875  mov         rcx,rax  
00007FF743771878  mov         rax,qword ptr [&action]  
00007FF74377187F  call        qword ptr [rax]  

But if I swap the two includes around, or remove the second argument from the function, I get a completely different disassembly:

    ( statesManager_->*action )( );
00007FF68CB01860  mov         rax,qword ptr [this]  
00007FF68CB01867  mov         rcx,qword ptr [rax]  
00007FF68CB0186A  call        qword ptr [action]  

The first code crashes on the call instruction. It attempts to read a dword value at &action+8, which has never been initialized, and results in a crash on the call instruction.

I found a related bug from half a year ago, but it's supposed to have been fixed in 15.9, when I'm currently on 15.9.7.

Is it another bug in VS2017 or I'm doing something unintended with member function pointers and forward declarations?

Answer 1

I am almost sure that the problem can be solved with the option /vmg.

Otherwise the compiler will optimize the representation of member pointers depending the class definition. A class with multiple base classes needs different member pointers than a class without them and a class with virtual base classes may need even more complex ones.

Without /vmg the compiler will generate different code depending on whether it has seen the full definition of IStatesManager, which by the name I assume to be an interface with virtual methods.

All modules using this class would have to be compiled with the /vmg option as well, so the right kind of member pointer gets passed in.

Alternatively, you could probably include the header for IStatemanager in the ControlNode header, but I assume the forward declaration was used deliberately to reduce dependencies.

Edit: The compiler still optimizes the method pointer calling code when it knows the class definition, and can thus rule out the complicated virtual derivation case, as stated in the comments the important difference is in the initialization of the method pointers, which is guaranteed to be consistent with /vmg.

The code generated for these functions shows the difference:

struct VirtMethods
{
  virtual int m();
};

struct VDerived : public virtual VirtMethods
{
  virtual int m() override;
};

int invokeit2(VirtMethods &o, int (VirtMethods::*method)());
int invokeit2(VDerived &o, int (VDerived::*method)());

int test(VirtMethods &o)
{
    return invokeit2(o, &VirtMethods::m);
}

int test(VDerived &o)
{
    return invokeit2(o, &VDerived::m);
}

Without /vmg, the following code is generated, that just passes a simple function pointer in a register for a class with just virtual methods. On the other hand a class with a virtual base class needs much more data in a struct passed in memory.

o$ = 8
int test(VirtMethods &) PROC                  ; test, COMDAT
        lea     rdx, OFFSET FLAT:[thunk]:VirtMethods::`vcall'{0,{flat}}' }' ; VirtMethods::`vcall'{0}'
        jmp     int invokeit2(VirtMethods &,int (__cdecl VirtMethods::*)(void)) ; invokeit2
int test(VirtMethods &) ENDP                  ; test

$T1 = 32
$T2 = 32
o$ = 64
int test(VDerived &) PROC               ; test, COMDAT
$LN4:
        sub     rsp, 56                             ; 00000038H
        and     DWORD PTR $T2[rsp+8], 0
        lea     rax, OFFSET FLAT:[thunk]:VDerived::`vcall'{0,{flat}}' }'     ; VDerived::`vcall'{0}'
        mov     QWORD PTR $T2[rsp], rax
        lea     rdx, QWORD PTR $T1[rsp]
        mov     DWORD PTR $T2[rsp+12], 4
        movaps  xmm0, XMMWORD PTR $T2[rsp]
        movdqa  XMMWORD PTR $T1[rsp], xmm0
        call    int invokeit2(VDerived &,int (__cdecl VDerived::*)(void)) ; invokeit2
        add     rsp, 56                             ; 00000038H
        ret     0
int test(VDerived &) ENDP               ; test


[thunk]:VDerived::`vcall'{0,{flat}}' }' PROC                         ; VDerived::`vcall'{0}', COMDAT
        mov     rax, QWORD PTR [rcx]
        jmp     QWORD PTR [rax]
[thunk]:VDerived::`vcall'{0,{flat}}' }' ENDP                         ; VDerived::`vcall'{0}'

[thunk]:VirtMethods::`vcall'{0,{flat}}' }' PROC                            ; VirtMethods::`vcall'{0}', COMDAT
        mov     rax, QWORD PTR [rcx]
        jmp     QWORD PTR [rax]
[thunk]:VirtMethods::`vcall'{0,{flat}}' }' ENDP    

With /vmg on the other hand, the code for the simple class looks completely different:

$T1 = 32
$T2 = 64
o$ = 112
int test(VirtMethods &) PROC                  ; test, COMDAT
$LN4:
        sub     rsp, 104                      ; 00000068H
        lea     rax, OFFSET FLAT:[thunk]:VirtMethods::`vcall'{0,{flat}}' }' ; VirtMethods::`vcall'{0}'
        mov     QWORD PTR $T1[rsp], rax
        lea     rdx, QWORD PTR $T2[rsp]
        xor     eax, eax
        mov     QWORD PTR $T1[rsp+8], rax
        movups  xmm0, XMMWORD PTR $T1[rsp]
        mov     DWORD PTR $T1[rsp+16], eax
        movsd   xmm1, QWORD PTR $T1[rsp+16]
        movaps  XMMWORD PTR $T2[rsp], xmm0
        movsd   QWORD PTR $T2[rsp+16], xmm1
        call    int invokeit2(VirtMethods &,int (__cdecl VirtMethods::*)(void)) ; invokeit2
        add     rsp, 104                      ; 00000068H
        ret     0
int test(VirtMethods &) ENDP                  ; test