UTF-8 text in MFC application that uses Multibyte character set

Question

I am working on an application which receives text encoded in UTF-8 and needs to display it on some MFC control. The application is build using MultiByte character set (MBCS) and let's assume this cannot change.

I was hoping that if I convert the text I receive from UTF-8 to wide char string, I would be able to display it correctly using the SetWindowTextW method. To try this, I used a toy app which reads the input from a file and sets the texts of my controls.

std::wstring utf8_decode(const std::string &str)
{
    if (str.empty()) return std::wstring();
    int size_needed = MultiByteToWideChar(CP_UTF8, 0, &str[0], (int)str.size(), NULL, 0);
    std::wstring wstrTo(size_needed, 0);
    MultiByteToWideChar(CP_UTF8, 0, &str[0], (int)str.size(), &wstrTo[0], size_needed);
    return wstrTo;
}

BOOL CAboutDlg::OnInitDialog()
{
    std::vector<std::string> texts;
    texts.resize(6);
    std::fstream f("D:\\code\\sample-utf8.txt", std::ios::in);
    for (size_t i=0;i<6;++i)
        std::getline(f, texts[i]);

    ::SetWindowTextW(GetDlgItem(IDC_BUTTON1)->m_hWnd, utf8_decode(texts[0]).c_str());
    ::SetWindowTextW(GetDlgItem(IDC_BUTTON2)->m_hWnd, utf8_decode(texts[1]).c_str());
    ::SetWindowTextW(GetDlgItem(IDC_BUTTON3)->m_hWnd, utf8_decode(texts[2]).c_str());
    ::SetWindowTextW(GetDlgItem(IDC_BUTTON4)->m_hWnd, utf8_decode(texts[3]).c_str());
    ::SetWindowTextW(GetDlgItem(IDC_BUTTON5)->m_hWnd, utf8_decode(texts[4]).c_str());
    ::SetWindowTextW(GetDlgItem(IDC_BUTTON6)->m_hWnd, utf8_decode(texts[5]).c_str());
    return TRUE;
}

Having built the toy-app with MBCS, I am not getting what I would like.

As soon as I build the app using unicode, it all works fine

Does this mean there is no hope of using unicode text for individual controls when I build with MBCS? If it is possible, can you give me any pointers? Thank you.

Answer 1

An MBCS app creates MBCS windows, which, generally speaking, are only going to be able to display text from a single code page, even if you use the wide string interface.

For an MBCS app, the wide string version of SetWindowTextW is essentially converting the wide string to MBCS, using the user's current locale (which has a default code page), and then passing that to the -A version of the function.

As you've seen with your "Unicode" experiement, you're doing the right thing in general, but you're limited by the fact that the app is MBCS.

Answer 2

I got this working, thanks to the suggestions of Adrian McCarthy and Joseph Willcoxson. I create the controls using the CreateWindowExW method, then setting the text using the SetWindowTextW. Below is sample code in case it 's of any help:

std::wstring utf8_decode(const std::string &str)
{
    if (str.empty()) return std::wstring();
    int size_needed = MultiByteToWideChar(CP_UTF8, 0, &str[0], (int)str.size(), NULL, 0);
    std::wstring wstrTo(size_needed, 0);
    MultiByteToWideChar(CP_UTF8, 0, &str[0], (int)str.size(), &wstrTo[0], size_needed);
    return wstrTo;
}

HWND CreateButtonW(int x, int y, int width, int height, HWND parent)
{
    HWND hwndButton = ::CreateWindowExW(
        WS_EX_CLIENTEDGE,
        L"BUTTON",  // Predefined class; Unicode assumed 
        L"", WS_TABSTOP | WS_VISIBLE | WS_CHILD | BS_DEFPUSHBUTTON,
        x, y, width, height, parent,
        NULL, // No menu.
        (HINSTANCE)GetWindowLong(parent, GWL_HINSTANCE),
        NULL);      // Pointer not needed.
    return hwndButton;
}

BOOL CAboutDlg::OnInitDialog()
{
    std::vector<std::string> texts;
    texts.resize(6);
    std::fstream f("D:\\code\\sample-utf8.txt", std::ios::in);
    for (size_t i=0;i<6;++i)
        std::getline(f, texts[i]);


    ::SetWindowTextW(GetDlgItem(IDC_BUTTON1)->m_hWnd, utf8_decode(texts[0]).c_str());
    ::SetWindowTextW(GetDlgItem(IDC_BUTTON2)->m_hWnd, utf8_decode(texts[1]).c_str());
    ::SetWindowTextW(GetDlgItem(IDC_BUTTON3)->m_hWnd, utf8_decode(texts[2]).c_str());
    ::SetWindowTextW(GetDlgItem(IDC_BUTTON4)->m_hWnd, utf8_decode(texts[3]).c_str());
    ::SetWindowTextW(GetDlgItem(IDC_BUTTON5)->m_hWnd, utf8_decode(texts[4]).c_str());
    ::SetWindowTextW(GetDlgItem(IDC_BUTTON6)->m_hWnd, utf8_decode(texts[5]).c_str());

    auto width  = [](RECT& r) { return r.right - r.left; };
    auto height = [](RECT& r) { return r.bottom - r.right; };

    RECT r;
    GetDlgItem(IDC_BUTTON1)->GetWindowRect(&r); ScreenToClient(&r);
    HWND hBtnWnd = CreateButtonW(r.right+20, r.top, width(r), height(r), m_hWnd);
    ::SetWindowTextW(hBtnWnd, utf8_decode(texts[0]).c_str());

    GetDlgItem(IDC_BUTTON2)->GetWindowRect(&r); ScreenToClient(&r);
    hBtnWnd = CreateButtonW(r.right+20, r.top, width(r), height(r), m_hWnd);
    ::SetWindowTextW(hBtnWnd, utf8_decode(texts[1]).c_str());

    GetDlgItem(IDC_BUTTON3)->GetWindowRect(&r); ScreenToClient(&r);
    hBtnWnd = CreateButtonW(r.right+20, r.top, width(r), height(r), m_hWnd);
    ::SetWindowTextW(hBtnWnd, utf8_decode(texts[2]).c_str());

    GetDlgItem(IDC_BUTTON4)->GetWindowRect(&r); ScreenToClient(&r);
    hBtnWnd = CreateButtonW(r.right+20, r.top, width(r), height(r), m_hWnd);
    ::SetWindowTextW(hBtnWnd, utf8_decode(texts[3]).c_str());

    GetDlgItem(IDC_BUTTON5)->GetWindowRect(&r); ScreenToClient(&r);
    hBtnWnd = CreateButtonW(r.right+20, r.top, width(r), height(r), m_hWnd);
    ::SetWindowTextW(hBtnWnd, utf8_decode(texts[4]).c_str());

    GetDlgItem(IDC_BUTTON6)->GetWindowRect(&r); ScreenToClient(&r);
    hBtnWnd = CreateButtonW(r.right+20, r.top, width(r), height(r), m_hWnd);
    ::SetWindowTextW(hBtnWnd, utf8_decode(texts[5]).c_str());

    return TRUE;
}

And the result - on the left the buttons as created by default, on the right when created using CreateWindowExW: