UTF-8 output on Windows console

Question

The following code shows unexpected behaviour on my machine (tested with Visual C++ 2008 SP1 on Windows XP and VS 2012 on Windows 7):

#include <iostream>
#include "Windows.h"

int main() {
    SetConsoleOutputCP( CP_UTF8 );
    std::cout << "\xc3\xbc";
    int fail = std::cout.fail() ? '1': '0';
    fputc( fail, stdout );
    fputs( "\xc3\xbc", stdout );
}

I simply compiled with cl /EHsc test.cpp.

Windows XP: Output in a console window is ü0ü (translated to Codepage 1252, originally shows some line drawing charachters in the default Codepage, perhaps 437). When I change the settings of the console window to use the "Lucida Console" character set and run my test.exe again, output is changed to 1ü, which means

• the character ü can be written using fputs and its UTF-8 encoding C3 BC
• std::cout does not work for whatever reason
• the streams failbit is setting after trying to write the character

Windows 7: Output using Consolas is ��0ü. Even more interesting. The correct bytes are written, probably (at least when redirecting the output to a file) and the stream state is ok, but the two bytes are written as separate characters).

I tried to raise this issue on "Microsoft Connect" (see here), but MS has not been very helpful. You might as well look here as something similar has been asked before.

Can you reproduce this problem?

What am I doing wrong? Shouldn't the std::cout and the fputs have the same effect?

SOLVED: (sort of) Following mike.dld's idea I implemented a std::stringbuf doing the conversion from UTF-8 to Windows-1252 in sync() and replaced the streambuf of std::cout with this converter (see my comment on mike.dld's answer).

Answer 1

I understand the question is quite old, but if someone would still be interested, below is my solution. I've implemented a quite simple std::streambuf descendant and then passed it to each of standard streams on the very beginning of program execution.

This allows you to use UTF-8 everywhere in your program. On input, data is taken from console in Unicode and then converted and returned to you in UTF-8. On output the opposite is done, taking data from you in UTF-8, converting it to Unicode and sending to console. No issues found so far.

Also note, that this solution doesn't require any codepage modification, with either SetConsoleCP, SetConsoleOutputCP or chcp, or something else.

That's the stream buffer:

class ConsoleStreamBufWin32 : public std::streambuf
{
public:
    ConsoleStreamBufWin32(DWORD handleId, bool isInput);

protected:
    // std::basic_streambuf
    virtual std::streambuf* setbuf(char_type* s, std::streamsize n);
    virtual int sync();
    virtual int_type underflow();
    virtual int_type overflow(int_type c = traits_type::eof());

private:
    HANDLE const m_handle;
    bool const m_isInput;
    std::string m_buffer;
};

ConsoleStreamBufWin32::ConsoleStreamBufWin32(DWORD handleId, bool isInput) :
    m_handle(::GetStdHandle(handleId)),
    m_isInput(isInput),
    m_buffer()
{
    if (m_isInput)
    {
        setg(0, 0, 0);
    }
}

std::streambuf* ConsoleStreamBufWin32::setbuf(char_type* /*s*/, std::streamsize /*n*/)
{
    return 0;
}

int ConsoleStreamBufWin32::sync()
{
    if (m_isInput)
    {
        ::FlushConsoleInputBuffer(m_handle);
        setg(0, 0, 0);
    }
    else
    {
        if (m_buffer.empty())
        {
            return 0;
        }

        std::wstring const wideBuffer = utf8_to_wstring(m_buffer);
        DWORD writtenSize;
        ::WriteConsoleW(m_handle, wideBuffer.c_str(), wideBuffer.size(), &writtenSize, NULL);
    }

    m_buffer.clear();

    return 0;
}

ConsoleStreamBufWin32::int_type ConsoleStreamBufWin32::underflow()
{
    if (!m_isInput)
    {
        return traits_type::eof();
    }

    if (gptr() >= egptr())
    {
        wchar_t wideBuffer[128];
        DWORD readSize;
        if (!::ReadConsoleW(m_handle, wideBuffer, ARRAYSIZE(wideBuffer) - 1, &readSize, NULL))
        {
            return traits_type::eof();
        }

        wideBuffer[readSize] = L'\0';
        m_buffer = wstring_to_utf8(wideBuffer);

        setg(&m_buffer[0], &m_buffer[0], &m_buffer[0] + m_buffer.size());

        if (gptr() >= egptr())
        {
            return traits_type::eof();
        }
    }

    return sgetc();
}

ConsoleStreamBufWin32::int_type ConsoleStreamBufWin32::overflow(int_type c)
{
    if (m_isInput)
    {
        return traits_type::eof();
    }

    m_buffer += traits_type::to_char_type(c);
    return traits_type::not_eof(c);
}

The usage then is as follows:

template<typename StreamT>
inline void FixStdStream(DWORD handleId, bool isInput, StreamT& stream)
{
    if (::GetFileType(::GetStdHandle(handleId)) == FILE_TYPE_CHAR)
    {
        stream.rdbuf(new ConsoleStreamBufWin32(handleId, isInput));
    }
}

// ...

int main()
{
    FixStdStream(STD_INPUT_HANDLE, true, std::cin);
    FixStdStream(STD_OUTPUT_HANDLE, false, std::cout);
    FixStdStream(STD_ERROR_HANDLE, false, std::cerr);

    // ...

    std::cout << "\xc3\xbc" << std::endl;

    // ...
}

Left out wstring_to_utf8 and utf8_to_wstring could easily be implemented with WideCharToMultiByte and MultiByteToWideChar WinAPI functions.