What explains this strange PeekMessage behaviour (trying to deal with a full message queue, filtering for specific messages)?

Question

Our application acts as a COM server where all automation occurs within a single STA apartment (in the application's main thread), and some VBS scripts which make lengthy (>10 minute) calls are failing with the error "System call failed (80010100)". Some research (one, two, three) indicates this is probably caused by the message queue filling up, so that when COM tries to invoke the next method it is unable to.

In case it's important, the app is developed with Embarcadero RAD Studio 2010 (mostly C++, smatterings of Delphi for some of the COM classes.)

I thought I would examine the thread's message queue at the end of the lengthy COM method call (i.e., just before it returns) to see what it contains, by using GetQueueStatus and PeekMessage. While it appears that the queue is full, I am seeing some odd behaviour and I'm having trouble figuring out both why PeekMessage is behaving the way it is, and exactly why the queue is full - i.e., what it's full with.

Slightly lengthy explanation ahead:

Testing the thread's message queue is full

Code like this:

int iMessages = 0;
DWORD dwThreadId = GetCurrentThreadId();
while (::PostThreadMessage(dwThreadId, WM_USER, 0, 0)) {
  iMessages++;
}
if (GetLastError() == ERROR_NOT_ENOUGH_QUOTA) {
  String strError = L"Not enough quota, posted " + IntToStr(iMessages) + L" messages";
  // Do something with strError
}

when run at the end of a short COM-invoked method can post thousands (say, 9996) messages; at the end of the lengthy method call that is causing the script to fail, it can post 0. My conclusion is that the message queue being full really is the cause of the problem. My system's message queue limit is the default 10000 (see the Remarks section.)

A call to Application->ProcessMessages() (invokes the app's message loop until it's empty, for those of you who aren't Delphi / C++Builder users - it's a fairly normal "get/translate/dispatch until no more messages" method) solves the problem and the COM script can invoke the next method successfully. Although probably okay in this specific situation, calling ProcessMessages() in effectively random spots is something to avoid - it can lead to re-entrancy. I'd like to find out what is causing the queue to be full if possible.

Examining the contents of the message queue

Using GetQueueStatus to determine what sort of messages are in the queue reveals that there are timer (QS_TIMER), posted messages (QS_POSTMESSAGE), 'all posted messages' (i.e. other posted ones, QS_ALLPOSTMESSAGE), and paint messages (QS_PAINT).

Here's where it gets weird. I'm trying to remove select messages or types of messages using PeekMessage with PM_REMOVE to both partially empty the queue and to count the number of each type of message.

If I call:

while (::PeekMessage(&oMsg, NULL, 0, 0, PM_REMOVE | PM_NOYIELD | (QS_TIMER << 16)) != 0) {...

I get just over ten thousand messages, usually 10006 or so. Not all of them are WM_TIMER: several thousand are WM_APP+202, a message we use internally, which does not appear to be being posted (by us) in anywhere near such huge quantities. I've checked this: it's only sent a few times. There are also a few thousand of another WM_APP+something message we use; this one probably genuinely is being sent too often.

If I call this instead :

while (::PeekMessage(&oMsg, NULL, WM_TIMER, WM_TIMER, PM_REMOVE | PM_NOYIELD) != 0) {...

I get about ten messages, all of which are truly WM_TIMERs. Why? The PeekMessage documentation indicates that passing QS_TIMER << 16 should process only timer messages, but it produces vastly more messages, many of which aren't timers at all.

Finally, if I call a third variation instead:

while (::PeekMessage(&oMsg, NULL, WM_APP+202, WM_APP+202, PM_REMOVE | PM_NOYIELD) != 0) {...

which is filtering directly for the custom message that the first line of code returns thousands of, I get seventeen messages removed.

I've reproduced all this several times - none of it is once-off behaviour.

So:

• Why does the first call to PeekMessage remove more than just timers (compared to the second call)? Curiosity only, really.
• Why does the first call to PeekMessage remove several thousand of the WM_APP+202 messages (one we define and use and don't send that many of) but if I instead call the third variation, which filters directly for that specific message, I get 17?
• If I get such differing results for the same message, how do I figure out what's filled up the queue and how best to avoid it?
• Or to avoid all the above: can I safely ignore all this, and so then how should I deal with a full message queue when COM is about to try invoking a method?

I'm puzzled, and may well be making an elementary mistake - it's got to the stage of looking at something puzzling where you do that. Any help for the COM problem or explanations of the message behaviour, including 'You made elementary mistake X, golly that was stupid of you', will be greatly appreciated :)

Answer 1

GetQueueStatus() accepts QS_xxx parameters but PeekMessage() accepts only PM_QS_xxx constants.

This explains the discrepancy between the number of WM_TIMER messages indicated by QueueStatus and subsequently removed by PeekMessage(). Your PeekMessage(PM_REMOVE) call is not removing WM_TIMER messages but something else entirely.

I think you have misunderstood the documentation of PeekMessage(). PM_QS_POSTMESSAGE is documented as being of equivalent value as:

((QS_POSTMESSAGE | QS_HOTKEY | QS_TIMER) << 16)

And other PM_QS_xxx constants are documented as being equal to the corresponding QS_xxx constant << 16, but nowhere does it say that this is consistently the case and can be extrapolated to ALL QS_xxxx constants.

I suspect that QS_TIMER << 16 is yielding some filter which is doing more than just filtering WM_TIMER messages (clearly it is, I just can't say with certainty what filter it yields).

As far as I know, WM_TIMER is the only timer related message so there is no need to have a wider filter for a larger super set of timer messages - there is no such super set. If you want to filter timer messages just filter WM_TIMER.