Why can't I redirect output from WriteConsole?

Question

In the following program I print to the console using two different functions

#include <windows.h>

int main() {
    HANDLE h = GetStdHandle(STD_OUTPUT_HANDLE);
    DWORD byteswritten;
    WriteConsole(h, "WriteConsole", 12, &byteswritten, NULL);
    WriteFile(h, "WriteFile", 9, &byteswritten, NULL);
}

If when I execute this program and redirect it's output using a > out.txt or a 1> out.txt nothing gets printed to the console (as expected) but the contents of out.txt are only

WriteFile

What is different between the two that allows calls to WriteFile to be redirected to the file and calls to WriteConsole to go to ... nowhere

Tested with gcc and msvc on windows 10

Answer 1

WriteConsole only works with console screen handles, not files nor pipes.

If you are only writing ASCII content you can use WriteFile for everything.

If you need to write Unicode characters you can use GetConsoleMode to detect the handle type, it fails for everything that is not a console handle.

When doing raw output like this you also have to deal with the BOM if the handle is redirected to a file.

This blog post is a good starting point for dealing with Unicode in the Windows console...

Answer 2

Edit 2021:

Windows 10 now has the ConPTY API (aka pseudo-console), which basically allows any program to act like the console for another program, thus enables capturing output that is directly written to the console.

This renders my original answer obsolete for Windows versions that support ConPTY.

---

Original answer:

From the reference:

WriteConsole fails if it is used with a standard handle that is redirected to a file. If an application processes multilingual output that can be redirected, determine whether the output handle is a console handle (one method is to call the GetConsoleMode function and check whether it succeeds). If the handle is a console handle, call WriteConsole. If the handle is not a console handle, the output is redirected and you should call WriteFile to perform the I/O.

This is only applicable if you control the source code of the application that you want to redirect. I recently had to redirect output from a closed-source application that unconditionally called WriteConsole() so it could not be redirected normally.

Reading the console screen buffer (as suggested by this answer) prooved to be unreliable, so I used Microsoft Detours library to hook the WriteConsole() API in the target process and call WriteFile() if necessary. Otherwise call the original WriteConsole() function.

I created a hook DLL based on the example of Using Detours:

#include <windows.h>
#include <detours.h>

// Target pointer for the uninstrumented WriteConsoleW API.
//
auto WriteConsoleW_orig = &WriteConsoleW;

// Detour function that replaces the WriteConsoleW API.
//
BOOL WINAPI WriteConsoleW_hooked(
  _In_             HANDLE  hConsoleOutput,
  _In_       const VOID    *lpBuffer,
  _In_             DWORD   nNumberOfCharsToWrite,
  _Out_            LPDWORD lpNumberOfCharsWritten,
  _Reserved_       LPVOID  lpReserved 
)
{
    // Check if this actually is a console screen buffer handle.
    DWORD mode;
    if( GetConsoleMode( hConsoleOutput, &mode ) )
    {
        // Forward to the original WriteConsoleW() function.
        return WriteConsoleW_orig( hConsoleOutput, lpBuffer, nNumberOfCharsToWrite, lpNumberOfCharsWritten, lpReserved );
    }
    else
    {
        // This is a redirected handle (e. g. a file or a pipe). We multiply with sizeof(WCHAR), because WriteFile()
        // expects the number of bytes, but WriteConsoleW() gets passed the number of characters.
        BOOL result = WriteFile( hConsoleOutput, lpBuffer, nNumberOfCharsToWrite * sizeof(WCHAR), lpNumberOfCharsWritten, nullptr );

        // WriteFile() returns number of bytes written, but WriteConsoleW() has to return the number of characters written.
        if( lpNumberOfCharsWritten )
            *lpNumberOfCharsWritten /= sizeof(WCHAR);
        
        return result;
    }
}

// DllMain function attaches and detaches the WriteConsoleW_hooked detour to the
// WriteConsoleW target function.  The WriteConsoleW target function is referred to
// through the WriteConsoleW_orig target pointer.
//
BOOL WINAPI DllMain(HINSTANCE hinst, DWORD dwReason, LPVOID reserved)
{
    if (DetourIsHelperProcess()) {
        return TRUE;
    }

    if (dwReason == DLL_PROCESS_ATTACH) {
        DetourRestoreAfterWith();

        DetourTransactionBegin();
        DetourUpdateThread(GetCurrentThread());
        DetourAttach(&(PVOID&)WriteConsoleW_orig, WriteConsoleW_hooked);
        DetourTransactionCommit();
    }
    else if (dwReason == DLL_PROCESS_DETACH) {
        DetourTransactionBegin();
        DetourUpdateThread(GetCurrentThread());
        DetourDetach(&(PVOID&)WriteConsoleW_orig, WriteConsoleW_hooked);
        DetourTransactionCommit();
    }
    return TRUE;
}

Note: In the WriteFile() branch I don't write a BOM (byte order mark), because it is not always wanted (e. g. when redirecting to a pipe instead of a file or when appending to an existing file). An application that is using the DLL to redirect process output to a file can simply write the UTF-16 LE BOM on its own before launching the redirected process.

The target process is created using DetourCreateProcessWithDllExW(), specifying the name of our hook DLL as argument for the lpDllName parameter. The other arguments are identical to how you create a redirected process via the CreateProcessW() API. I won't go into detail, because these are all well documented.