PeekMessage triggers WndProc callback

Question

Yesterday I encountered the weirdest problem I have ever seen. I wrote a module that should get a notification on USB plugs. To do so, I created a dummy window and registered it to device change notifications using some interface's GUID.

The strange error occurs when PeekMessage is called. at this point, some why, the window's WndProc callback is called, only when the peeked message is WM_DEVICECHANGE (we were registered to in the above code). On any other message, the DispatchMessage triggers the callback, as expected.

code:

NotificationFilter.dbcc_size = sizeof(DEV_BROADCAST_DEVICEINTERFACE);
NotificationFilter.dbcc_devicetype = DBT_DEVTYP_DEVICEINTERFACE;
NotificationFilter.dbcc_classguid = guid;
not = RegisterDeviceNotification(
    hWnd,     // events recipient
    &NotificationFilter,        // type of device
    DEVICE_NOTIFY_WINDOW_HANDLE // type of recipient handle
);

In order to incorporate this module with the rest of my code which is asynchronous, using Reactor design pattern with Windows Events, and following the advice of stackoverflow community members, I incorporated MsgWaitForMultipleObjects in order to listen for events and windows messages as well.

code:

for (;;)
{
    dwRetval = MsgWaitForMultipleObjects(cntEvents, arrEvents, FALSE, INFINITE, QS_ALLINPUT);
    switch (dwRetval)
    {
    case WAIT_FAILED:
        // failed. TODO: status
        break;
    // TODO: handle abandoned.
    default:
        if (dwRetval == cntEvents)
        {
            // Message has popped.
            BOOL x = PeekMessage(&tMsg, hWnd, 0, 0, PM_REMOVE); <---- WM_DEVICECHANGE triggers the callback
            if (x)
            {
                TranslateMessage(&tMsg);
                DispatchMessage(&tMsg);
            }
        }
        else if (dwRetval < cntEvents)
        {
            // event signaled
        }
        else
        {
            // TODO: status. unexpected.
            return FALSE; // unexpected failure
        }
        break;
    }
}

I disassembled the code, and compared the registers before any call to NtUserPeekMessage

Registers on successful call:

RAX = 00000059A604EFB0 RBX = 0000000000000000 RCX = 00000059A604EF18
RDX = 0000000000070C62 RSI = 00000059A604EF18 RDI = 0000000000070C62
R8  = 0000000000000000 R9  = 0000000000000000 R10 = 00007FF71A65D800
R11 = 0000000000000246 R12 = 0000000000000000 R13 = 0000000000000000
R14 = 0000000000000000 R15 = 0000000000000000 RIP = 00007FF954562AA1
RSP = 00000059A604EE70 RBP = 0000000000000000 EFL = 00000200 

Registers on unknown callback trigger call:

RAX = 00000059A604EFB0 RBX = 0000000000000000 RCX = 00000059A604EF18
RDX = 0000000000070C62 RSI = 00000059A604EF18 RDI = 0000000000070C62
R8  = 0000000000000000 R9  = 0000000000000000 R10 = 00007FF71A65D800
R11 = 0000000000000246 R12 = 0000000000000000 R13 = 0000000000000000
R14 = 0000000000000000 R15 = 0000000000000000 RIP = 00007FF954562AA1
RSP = 00000059A604EE70 RBP = 0000000000000000 EFL = 00000200 

The registers are exactly the same! (No parameters are passed on the stack, 64bit..)

In both cases (strange error and expected flow) I stepped into at NtUserPeekMessage, it turns out that the WndProc callback is triggered only from the internal syscall!

00007FF954562A80  mov         r10,rcx  
00007FF954562A83  mov         eax,1003h  
00007FF954562A88  syscall  

I couldn't find any documentation on MSDN or explanation on the internet to the phenomenon.

I would really like some help, Thanks in advance.

Answer 1

That is as expected, and is documented. PeekMessage is one of the functions that dispatches sent messages. From the documentation:

Dispatches incoming sent messages, checks the thread message queue for a posted message, and retrieves the message (if any exist).

And then later in the same document:

During this call, the system delivers pending, nonqueued messages, that is, messages sent to windows owned by the calling thread using the SendMessage, SendMessageCallback, SendMessageTimeout, or SendNotifyMessage function.

The documentation for SendMessage says this (with my emphasis):

If the specified window was created by the calling thread, the window procedure is called immediately as a subroutine. If the specified window was created by a different thread, the system switches to that thread and calls the appropriate window procedure. Messages sent between threads are processed only when the receiving thread executes message retrieval code.

By message retrieval code the documentation means functions like GetMessage and PeekMessage. There are a few others, I don't have a comprehensive list at hand.