Why is the /utf-8 flag in MSVC not allowing my program to display Unicode characters?

Question

I recently discovered that on Windows 10/11 there is a beta testing option under region settings (system locale) to "Use Unicode UTF-8 for worldwide language support". When this is enabled, all the ANSI Win32 system calls treat string as UTF-8. Sure enough, if enabled you can compile the following in MSVC:

int main() {
    std::cout << "Hello, World! こんにちは世界!" << std::endl;
    //prints "Hello, World! こんにちは世界!"
}

I then read that you don't have to enable this system-wide and can instead compile your program with the /utf-8 flag. So with the beta option disabled and the /utf-8 flag added to my project:

int main() {
    std::cout << u8"Hello, World! こんにちは世界!" << std::endl;
    //prints "Hello, World! こんにちは世界!"
}

and

int main() {
    setlocale(LC_ALL, "en_US.utf-8");
    std::cout << "Hello, World! こんにちは世界!" << std::endl;
    //prints "Hello, World! ???????!"
}

I also tried adding u8 to the string literal, but it makes no difference.

Answer 1

C++23 std::print supports portable Unicode output. On Windows/MSVC all you need is to compile with /utf-8:

#include <print>

int main() {
  std::print("Hello, World! こんにちは世界!\n");
}

This prints

Hello, World! こんにちは世界!

If it is not yet available then you can use the {fmt} library, std::print is based on:

#include <fmt/core.h>

int main() {
  fmt::print("Hello, World! こんにちは世界!\n");
}

Unfortunately, due to backward compatibility reasons std::cout is unlikely to be changed to support Unicode on Windows in a similar way. In your case std::cout produces mojibake because the ordinary literal encoding doesn't match the terminal encoding which is a pretty common issue.

Disclaimer: I am the author of C++23 std::print and {fmt}.

Answer 2

Use #pragma execution_character_set("utf-8") and SetConsoleOutputCP(CP_UTF8), eg:

#include<iostream>

#include <Windows.h>
#pragma execution_character_set("utf-8")

int main() {
    SetConsoleOutputCP(CP_UTF8);
    std::cout << "Hello, World! こんにちは世界!" << std::endl;       
}

Update:

As Remy Lebeau said, you can use app manifest.

yourapp.manifest:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly manifestVersion="1.0" xmlns="urn:schemas-microsoft-com:asm.v1">
  <assemblyIdentity type="win32" name="Microsoft.Windows.Common-Controls" version="6.0.0.0"/>
  <application>
    <windowsSettings>
      <activeCodePage xmlns="http://schemas.microsoft.com/SMI/2019/WindowsSettings">UTF-8</activeCodePage>
    </windowsSettings>
  </application>
</assembly>

Add the manifest in Visual Studio Project properties -> Manifest Tools -> Input and Output -> Additional Manifest Files: yourapp.manifest

Or Visual Studio Command Prompt :

mt.exe -manifest yourapp.manifest -outputresource:yourapp.exe;#1