Strange behaviour with Japanese IME in Windows 10 1709

Question

I am facing a problem in Windows 10 Creators update where, when I try to input something to my application using IME, the first character is ignored; i.e., If I use IME to enter the japanese hiragana character 'か' by typing K & A, i end up getting only 'あ' with the K being lost. This happens to the first character only. But the exact same application works properly in Windows 7~8.

The details are as below:

The application is an MFC MDI application of the Container/Server type. Its working is really simple & straightforward. If a document is open, then when a WM_KEYDOWN is fired, dynamically create a CEdit box and enter the pressed key into the edit box. If the edit box already exists, no need to create it again. Just append the input to the contents of the edit box.

I created 2 sample MFC MDI projects (e.g. MDI_sample1 & MDI_Sample2). Keeping the default cpp & h files as is, just added a new class (e.g. CwEdit) that subclasses the CEdit class to both MDI_Sample1 & MDI_Sample2 projects. Now, in MDI_Sample1, I open the *View.cpp, and add a WindowProc override. In this function, I check for the WM_KEYDOWN message, and on WM_KEYDOWN excepting VK_BACK, VK_ENTER, VK_TAB, I dynamically create an edit box using the CwEdit class, and then SendMessage a WM_KEYDOWN with the current wParam and lParam that I got as arguments of the WindowProc function. Running the program, I create a document and then press the key 'k'. An edit box will get created in the document. If IME is not being used, the character 'k' will also get entered into this newly created edit box. Next,I press 'a'and the character 'a' is appended to 'k' in the edit box. So far so good.

Next, I create a new document again. This time, I activate the windows IME to japanese and input 'k'. Again, an edit box will get created and it will display the 'k' with wavy underlines. I input 'a' and it correctly displays the japanese character 'か'. Again, expected and correct.

I copy this exe file to a windows 10 1709 machine and run it. Again, I repeat the same steps above to input the character 'k'. Without IME being active, the box is created and the 'k' is entered into it. Next I press 'a' and the edit box will correctly read 'ka'. Next, I create a new document. This time, I activate the windows IME to japanese and input 'k'. Again, an edit box will get created but it will be empty. I input 'a' and it now displays the japanese character 'あ'. This behaviour happens to all characters. The first keydown which was used to create the edit box will not be shown when IME is active. But once the edit box is created, everything works fine.

I copy the whole code to MDI_Sample2. But there is one little change. This time, in the view, I override the PreTranslateMessage and do the exact same process which was previously being done inside WindowProc. And remove the WindowProc override. This MDI_Sample2 works perfectly well both on Windows 7 as well as Windows 10 1709 even when the Japanese IME is active.

The code for the *View.cpp for both the projects are as given below:

MDI_Sample1View.cpp

---

BOOL MDI_Sample1View::WindowProc(UINT message, WPARAM wParam, LPARAM lParam)
{
    // TODO: Add your specialized code here and/or call the base class
    if(message == WM_CHAR)
    {
        int wp = static_cast<int>(wParam);
        // All printable ascii characters
        if (wp >= 0x32 && wp <= 0x255)
        {
            EnableEdit();
            M_pEdit->SendMessage(message, wParam, lParam);
            return TRUE;
        }
    }
    else if(message == WM_KEYDOWN)
    {
        if (wParam == VK_ESCAPE)
        {
            if(M_pEdit &&
                GetFocus() == M_pEdit)
            {
                DisableEdit();
                return TRUE;
            }
        }
        EnableEdit();
    }
    return CView::WindowProc(message, wParam, lParam);
}

MDI_Sample2View.cpp

---

BOOL MDI_Sample2View::PreTranslateMessage(MSG* pMsg)
{
    // TODO: Add your specialized code here and/or call the base class
    if(pMsg->message == WM_CHAR)
    {
        int wp = static_cast<int>(pMsg->wParam);
        // All printable ascii characters
        if (wp >= 0x32 && wp <= 0x255)
        {
            EnableEdit();
            M_pEdit->SendMessage(pMsg->message, pMsg->wParam, pMsg->lParam);
            return TRUE;
        }
    }
    else if(pMsg->message == WM_KEYDOWN)
    {
        if (pMsg->wParam == VK_ESCAPE)
        {
            if(M_pEdit &&
                GetFocus() == M_pEdit)
            {
                DisableEdit();
                return TRUE;
            }
        }
        EnableEdit();
    }
    return CView::PreTranslateMessage(pMsg);
}

All the other files are the same as created by visual studio when I created the new project. The CwEdit.cpp class has 2 functions namely Create to create the edit box, and an OnKeyDown which is given below:

void CwSpEdit::OnKeyDown(UINT nChar, UINT nRepCnt, UINT nFlags) 
{
    if(nChar == VK_ESCAPE)
    {
        SetWindowText(_T(""));
        return;
    }
    CEdit::OnKeyDown(nChar, nRepCnt, nFlags);
}

Rest of both the projects are identical. So, what is happening here? Why does WindowProc disregard the first character while PreTranslateMessage works fine?

How do I solve this issue? I need to make it work with WindowProc as it used to.

Update:

Some additional details on the issue. As an example, I try to input the Japanese word 'さくら'. Using english alphabets, this would be spelled as 'sakura'. Now, I run the application, select the Microsoft IME for japanese hiragana input and type 'sakura'. Upto windows 10 prior to the Creators update, this will work as follows. The 's' keystroke will generate the edit box. Following this, it will also invoke the IME composition window which will now display 's' with wavy underlines. the following keystroke 'a' will update the 's' in the IME window to the japanese character 'さ'. The next keystroke 'k' will update the IME window to show 'さk' with the k having wavy underlines and so on. This is the expected and correct behaviour.

In Windows 10 1709, how it works is: the first keystroke of 's' will generate the edit box. But no IME composition window shows up. No error or warning messages are shows even during debug runs. The next keystroke 'a' will now invoke the IME composition window with the japanese equivalent of 'a' that is the character 'あ'. Meaning, finally, I get 'あくら' which in english alphabets is 'akura'. The first 's' is lost.

This happens when I handle the edit box creation using 'WindowsProc'. In this case, it will work correctly until you update your OS to Windows 10 1709. On the other hand, if I create the edit box in 'PreTranslateMessage' it will work correctly even in Windows 10 1709. What changed in the way 'WindowsProc' is handled in Windows 10 1709 and How to work around it?

Answer 1

Finally I figured things out. It seems that the IME behaviour has changed in Windows 10 1709 and above. I will explain the different behaviours with an example:

Case 1: Prior to Windows 10 1709 -> Open notepad. Set IME to Japanese Hiragana and press the 'k' key. You will see a 'k with a wavy underline'. You need one or more characters to compose this 'k' to a proper hiragana. Until you provide more input, or press Esc to cancel the input, the 'k' will remain as an unconfirmed IME input. Leaving it as it is without any additional inputs, just click somewhere else (like your desktop) such that notepad looses focus. You will note that the IME indicator in your taskbar/languagebar has changed. And you MIGHT also see windows' own IME composition window (the small black one in windows 7) pop up with your 'k' in it. Now focus back to notepad and you will find that the unconfirmed 'k' is still hanging around waiting for you to either provide further input or cancel it. In short, when focus is changed, the unconfirmed IME strings are still kept as they are in the unconfirmed state.

Case 2: Windows 10 1709 onwards: -> Repeat the above steps. here you can notice the difference. As soon as the focus changes, the IME composition is stopped. So, the 'k' being an UNCONFIRMED IME string gets discarded.

In the example given in the question, what was happening with WindProc and PreTranslateMessage was that, with WindProc, On IME keyPress, the View is under focus and recieves the KeyPress message. It processes it and passes it down to its children, where as per our code, a new edit control is created. Now, as the edit control is created, it gets focused. And according to the new behaviour, when the focus changes from the View to the control, IME composition is stopped. As this happens at the first IME keypress, we have got a character still in its unconfirmed state. Being an unconfirmed IME character, this character is discarded.

With PreTranslateMessage, we receive the keypress message and proceed to create the edit control. On creation, the edit control gets focus and our View looses focus. This generates the KillFocus message, but as we are still in the middle of the previous KeyPress message processing, the KillFocus message is not being processed. It is waiting for the completion of the previous message processing. Now, when we return back after creating the control, we pass on the keypress to the newly created editbox. Thus, it is the editbox that finally receives the keypress as well the following unconfirmed IME character. Thus, as per our example 'k' keypress, the editbox and NOT the view, receives the unconfirmed 'k'. The next keypress naturally is received by the editbox as it is under focus now and so this second input is added to the unconfirmed 'k' and composition is performed as usual.

This behaviour is not restricted to only characters needing multiple keypresses. Even single keypress characters like 'a' also act the same way because even these characters remain unconfirmed until we either press the Enter key, or select one of the composition candidates and so on.