Why is my TB_INSERTBUTTON message causing comctl32 to throw?

Question

I'm trying to add an additional button into a toolbar in Internet Explorer.

I assumed the implementation to be straight forward, and am currently using this code:

TBBUTTON buttonToAdd;
ZeroMemory( &buttonToAdd, sizeof( TBBUTTON ) );
buttonToAdd.iBitmap = 1;
buttonToAdd.idCommand = 1;
buttonToAdd.fsState = TBSTATE_ENABLED;
buttonToAdd.fsStyle = BTNS_BUTTON|BTNS_AUTOSIZE;

LRESULT insertButtonResult = SendMessage( hWndToolbar, TB_INSERTBUTTON, 0, (LPARAM)&buttonToAdd );

When the message is sent, Internet Explorer will crash 90% of the time (10% of the time, I get a somewhat broken button on the toolbar) with the following exception:

Unhandled exception at 0x000007FEFB97DDFA (comctl32.dll) in iexplore.exe: 0xC000041D: An unhandled exception was encountered during a user callback.

Given that the results aren't consistent, I assumed some sort of memory layout issue. So I tried to send TB_INSERTBUTTONA instead (my application defaults to TB_INSERTBUTTONW), but that has no effect on the issue.

I also tried both 32 and 64 builds of my application, both have the same result.

I took a look at the callstack of iexplore.exe, which looks like this:

comctl32.dll!CToolbar::TBInputStruct(struct _TBBUTTONDATA *,struct _TBBUTTON const *)   Unknown
comctl32.dll!CToolbar::TBInsertButtons(unsigned int,unsigned int,struct _TBBUTTON *,int)    Unknown
comctl32.dll!CToolbar::ToolbarWndProc(struct HWND__ *,unsigned int,unsigned __int64,__int64)    Unknown
comctl32.dll!CToolbar::s_ToolbarWndProc(struct HWND__ *,unsigned int,unsigned __int64,__int64)  Unknown
user32.dll!UserCallWinProcCheckWow()   Unknown
user32.dll!DispatchClientMessage() Unknown
user32.dll!__fnDWORD() Unknown
ntdll.dll!KiUserCallbackDispatcherContinue()   Unknown
user32.dll!NtUserPeekMessage() Unknown
user32.dll!PeekMessageW()  Unknown
...

I found that somewhat interesting, because I'm assuming the method at the top copies data from my input structure into an internal structure and something goes wrong. But what's wrong with my input data structure?

The source code itself is available on GitHub at: https://github.com/oliversalzburg/ie-button

Answer 1

It is failing because you are sending a message that contains a pointer across a process boundary. Note the fact that you pass an address:

LRESULT insertButtonResult = SendMessage(hWndToolbar, TB_INSERTBUTTON, 0, 
    (LPARAM)&buttonToAdd);

That final parameter is an address in your processes address space. But the recipient is a different process and the address you pass has no meaning in the address space of the other process.

Some messages, for instance WM_SETTEXT, will have their payloads marshalled to the other process by the system. But TB_INSERTBUTTON does not fall into that category. One of the rules of TB_INSERTBUTTON is that the pointer you pass has meaning in the process that owns the recipient window.

You can resolve this by using VirtualAlloc, WriteProcessMemory, etc. to allocate and write to memory in that other process.

Be warned that this is a somewhat difficult task to get right. In particular it matters greatly whether or not the two processes have the same bitness. The layout of the struct differs between 32 and 64 bit. The easiest way for you to make sure you send the right layout is to compile your process with the same bitness as the target process.

By far the easiest way to do something like this is to be inside the target process. If you were to write a plugin then you would not have to deal with any of these issues and would also be able to use officially supported APIs for extension.

As Raymond says, what you are attempting is rather dangerous and you would do well to heed his advice.