Why no text colors after using CreateFile("CONOUT$" ...) to redirect the console back to the active screen buffer?

Question

I had an old app based off of this setup (but more advanced): https://stackoverflow.com/a/33441726/1236397

Basically, it runs, processes stuff, and closes, and only opens a window if it needs to (runs via another application). Since porting this to Visual Studio 2017 on Windows 10 the output window failed to work. After a bit more investigation it appears the issue was related to the standard output being redirected for non-console applications (output type must be "Windows Application" to initially hide all windows). In this scenario, while debugging, there is no text output, which requires resetting the redirected output. This is done using CreateFile("CONOUT$", ...) to get and redirect the console back to the display buffer output.

[DllImport("kernel32.dll", CharSet = CharSet.Auto)] //, EntryPoint = "CreateFileW", SetLastError = true, CharSet = CharSet.Auto, CallingConvention = CallingConvention.StdCall
private static extern IntPtr CreateFileW(
      string lpFileName,
      UInt32 dwDesiredAccess,
      UInt32 dwShareMode,
      IntPtr lpSecurityAttributes,
      UInt32 dwCreationDisposition,
      UInt32 dwFlagsAndAttributes,
      IntPtr hTemplateFile
     );

const UInt32 GENERIC_WRITE = 0x40000000;
const UInt32 GENERIC_READ = 0x80000000;
const UInt32 OPEN_EXISTING = 0x00000003;

const int STD_INPUT_HANDLE = -10;
const int STD_OUTPUT_HANDLE = -11;
const int STD_ERROR_HANDLE = -12;

var outFile = CreateFileW("CONOUT$", GENERIC_WRITE, FILE_SHARE_WRITE, IntPtr.Zero, OPEN_EXISTING, /*FILE_ATTRIBUTE_NORMAL*/0, IntPtr.Zero);
var safeHandle = new SafeFileHandle(outFile, true);
SetStdHandle(STD_OUTPUT_HANDLE, outFile);
var fs = new FileStream(safeHandle, FileAccess.Write);
var writer = new StreamWriter(fs) { AutoFlush = true };
Console.SetOut(writer);

This works; however, there is no color support. How can colors be enabled?

Answer 1

As it turns out, after much digging, it seems there's a mode that does not get set when the application runs in "Windows Application" mode:

ENABLE_VIRTUAL_TERMINAL_INPUT (doc)

This is how to set it:

if (GetConsoleMode(outFile, out var cMode))
    SetConsoleMode(outFile, cMode | ENABLE_VIRTUAL_TERMINAL_INPUT);

However, it still would not work, and this is where my issue was. That if statement will ALWAYS fail because only GENERIC_WRITE was specified for CreateFileW()! Turns out GENERIC_READ ALSO needs to be set, otherwise, you cannot read the current mode. The CreateFileW line should be this:

var outFile = CreateFileW("CONOUT$", GENERIC_WRITE | GENERIC_READ, FILE_SHARE_WRITE, IntPtr.Zero, OPEN_EXISTING, /*FILE_ATTRIBUTE_NORMAL*/0, IntPtr.Zero);

My final code is different from this, but I wrapped up an example here for others if needed:

var outFile = CreateFileW("CONOUT$", GENERIC_WRITE | GENERIC_READ, FILE_SHARE_WRITE, IntPtr.Zero, OPEN_EXISTING, /*FILE_ATTRIBUTE_NORMAL*/0, IntPtr.Zero);
var safeHandle = new SafeFileHandle(outFile, true);
SetStdHandle(STD_OUTPUT_HANDLE, outFile);
var fs = new FileStream(safeHandle, FileAccess.Write);
var writer = new StreamWriter(fs) { AutoFlush = true };
Console.SetOut(writer);
if (GetConsoleMode(outFile, out var cMode))
    SetConsoleMode(outFile, cMode | ENABLE_VIRTUAL_TERMINAL_INPUT);

A lot of time was wasted trying to figure this out from info scattered everywhere, so I hope this helps save someone else time. ;)