How to eliminate the MessageBeep from the RICHEDIT control?

Question

The RichEdit control has this very annoying feature. It beeps every time the user tries to move the cursor past its "end point". For instance, you can test it with the WordPad that also implements RICHEDIT. Open it up, type in some text, then hit the Home key. If the cursor was not in the beginning of the line:

hitting Home key will move it there, but then hitting the Home key again will produce this beep.

At first glance it seemed like overriding WM_KEYDOWN and WM_KEYUP messages and blocking the situations when RICHEDIT can produce that beep was a solution ... until I actually started implementing it. Unfortunately though, it's not as simple as it sounds, as that control beeps in a LOT OF CASES! So my keystroke blocking code literally ballooned to over 300+ lines, and I still see that there are some key-presses that I either didn't account for, or worse, I might have overridden some useful behavior with. (Read below for more details.)

---

Then I decided to look inside the implementation of the RICHEDIT control itself. And sure enough, for instance if we look at the implementation of the Home key press, the C:\WINDOWS\SysWOW64\msftedit.dll on my Windows 10 OS, has the function called ?Home@CTxtSelection@@QAEHHH@Z (or public: int __thiscall CTxtSelection::Home(int,int) demangled) at the mapped offset 0x3FC00, that is hard-coded to call the MessageBeep(MB_OK), or exactly what I'm trying to eliminate:

And if you look at the address 0x6B64FD38 in the screenshot above, there's a built-in way to bypass it, with what looks to be flag 0x800.

So having dug into msftedit.dll a little bit more, there appears to be a function called ?OnAllowBeep@CTxtEdit@@QAEJH@Z (or public: long __thiscall CTxtEdit::OnAllowBeep(int) demangled) that can modify this flags:

After a bit more research I found out that there are COM interfaces built into RICHEDIT control, such as ITextServices and ITextHost that reference that flag as TXTBIT_ALLOWBEEP in ITextServices::OnTxPropertyBitsChange method.

Unfortunately though, I can't seem to find the way how I can directly change that TXTBIT_ALLOWBEEP flag (COM is not my forte.) I tried looking into implementing ITextHost, but it has a lot of virtual methods that have nothing to do with what I'm trying to achieve that I don't know how to implement.

Does anyone have any idea how to clear that TXTBIT_ALLOWBEEP flag?

---

PS. Here's why I didn't go the route of overriding key-presses: Just to give you an example. Say, if I override the VK_HOME key press. I need to make sure that the cursor is not at the beginning of the line, but also that there's no selection. Yet, I need to make sure that Ctrl key is not down in a situation when the cursor is at the very top of the window. Then the same with the Shift key, and I'm not even sure what Alt does with it ... and so forth. Oh, and this is just the Home key. There's also Up, Down, Left, Right, PageUp, PageDown, End, Delete, Backspace. (And that's what I was aware of. There may be more, plus I'm not even talking about IME or other keyboard layouts, etc.) In other words, it becomes a mess! So, eventually I realized that anticipating a keystroke is not the way to go.

Answer 1

first we need send EM_GETOLEINTERFACE message to rich edit window - this is Retrieves an IRichEditOle object that a client can use to access a rich edit control's Component Object Model (COM) functionality.

then for retrieve an ITextServices pointer, call QueryInterface on the private IUnknown pointer returned by EM_GETOLEINTERFACE.

here exist interesting point - the IID_ITextServices not well known but need get in runtime from Msftedit.dll

from About Windowless Rich Edit Controls

Msftedit.dll exports an interface identifier (IID) called IID_ITextServices that you can use to query the IUnknown pointer for the ITextServices interface.

after we got ITextServices pointer - we simply can call OnTxPropertyBitsChange with TXTBIT_ALLOWBEEP mask

code example:

#include <textserv.h>

if (HMODULE hmodRichEdit = LoadLibrary(L"Msftedit.dll"))
{
    // create richedit window
    if (HWND hwndRich = CreateWindowExW(0, MSFTEDIT_CLASS, ...))
    {
        if (IID* pIID_ITS = (IID*) GetProcAddress(hmodRichEdit, "IID_ITextServices"))
        {
            IUnknown* pUnk;
            if (SendMessageW(hwndRich, EM_GETOLEINTERFACE, 0, (LPARAM)&pUnk))
            {
                ITextServices* pTxtSrv;
                HRESULT hr = pUnk->QueryInterface(*pIID_ITS, (void**)&pTxtSrv);
                pUnk->Release();
                if (0 <= hr)
                {
                    pTxtSrv->OnTxPropertyBitsChange(TXTBIT_ALLOWBEEP, 0);
                    pTxtSrv->Release();
                }
            }
        }
    }
}