ENABLE_VIRTUAL_TERMINAL_PROCESSING and DISABLE_NEWLINE_AUTO_RETURN failing

Question

The point was to make this little multiplayer game in the terminal, applying some basic graphics concepts to get a grasp on how it works and the maths behind it.

Note I wish to do this for fun and I am fully aware there are way better alternatives out there to using a terminal.

I'd need a console I could write to, so the point was to remove scroll bars and have the whole buffer printed to the screen.

But because of the carriage return when characters get written to the end of the previous line:

this would look overall ugly:

After trying for about 3 hours to get this working through SetConsoleMode, I asked about it in a discord.

The answer I got was this:

void main( )
{
    auto h = GetStdHandle( STD_OUTPUT_HANDLE );

    DWORD mode = 0;
    GetConsoleMode( h, &mode );

    mode |= DISABLE_NEWLINE_AUTO_RETURN | ENABLE_VIRTUAL_TERMINAL_PROCESSING;

    puts( SetConsoleMode( h, mode ) ? "Win" : "Loss" );

    char * buf = new char[ 200*2 ];
    memset( buf, 0, 200*2 );
    memset( buf, 'A', 120*2 );
    std::cout << ( buf );

    getchar( );
}

with this result:

At first I tried adapting it to my project, which failed. Then I got frustrated and ended up creating a new project into which I just copy pasted the working code.

And you guessed it, it doesn't work.

Now I've tried quite a bit of stuff, I've changed my program from unicode to ansi and back, and done mode = DISABLE_NEWLINE_AUTO_RETURN | ENABLE_VIRTUAL_TERMINAL_PROCESSING, mode |= DISABLE_NEWLINE_AUTO_RETURN | ENABLE_VIRTUAL_TERMINAL_PROCESSING, and each individually with and without |with no success.

Given it's error 87 which means bad parameter, I even tried putting the exact value from GetConsoleMode to see if it was the handle, but since it worked we can assume it's not the handle's problem.

Basically code that works on another machine doesn't work on mine. The other machine was windows 10 mine's 8.1 both 64.

On both machines the value gotten off GetConsoleMode the first time is 3, which means ENABLE_PROCESSED_OUTPUT and ENABLE_WRAP_AT_EOL_OUTPUT.

The windows SDK version on my project is 10.0.15063.0, with a platform toolset of Visual Studio 2017 (v141).

I've been at this all day with no success. Now it wouldn't be the first time I read over some important detail, but I've been going through documentation and nothing mentions the failiure of SetConsoleMode using ENABLE_VIRTUAL_TERMINAL_PROCESSING.

As far as I can tell I'm doing it correctly based on the Docs

What should I be doing that I am not?

Sorry for any spelling/grammar mistakes that might have gone noticed and thank you for your time.

Answer 1

Virtual terminal mode is available in the console starting with Windows 10.0.10586. It's not supported by the OS if setting the mode fails with ERROR_INVALID_PARAMETER (87). Also, it's only implemented in the new console. With the legacy console selected in Windows 10, enabling VT mode may succeed, but it won't actually enable VT support.

The failure case is mistakenly documented as "SetConsoleMode returning with STATUS_INVALID_PARAMETER" (0xC000000D) in the Enabling Virtual Terminal Processing example. It seems the writer overlooked that the kernel status code that's returned by the NtDeviceIoControlFile system call (in Windows 8+) gets translated to failing the SetConsoleMode call and translating the status code to a Windows API error code.