SetStdHandle has no effect on cout/printf

Question

The title says it all. When I run the following code:

HANDLE hOut = GetStdHandle(STD_OUTPUT_HANDLE);
HANDLE hFile = CreateFile(TEXT("Foo.txt"), GENERIC_WRITE, FILE_READ_ACCESS | FILE_WRITE_ACCESS,
    NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);

SetStdHandle(STD_OUTPUT_HANDLE, hFile);
std::cout << "Hello, ";
printf("world!\n");
WriteFile(GetStdHandle(STD_OUTPUT_HANDLE), "Hello, world!\n", 13, NULL, NULL);

SetStdHandle(STD_OUTPUT_HANDLE, hOut);
CloseHandle(hFile);

The result is that Hello, world! gets written to the console as a result of the calls to cout and printf, and Hello, world! also gets written to the file Foo.txt as a result of the call to WriteFile. My assumption is that when everything gets initialized at the very beginning, the HANDLE returned by GetStdHandle is cached and reused for both cout and printf. That's perfectly reasonable and exactly what I would want as I assume GetStdHandle requires a call to the operating system (which can be long!). The trouble is that I want to override that behavior and "sync" both cout and printf with the application's standard handles if possible.

Before suggesting any alternatives, let me describe exactly what it is I am trying to do (yes, I know that it is possible to use freopen for this purpose). What I need to be able to do is to "save" the current standard output handle on a stack-like data structure before I change it so that I can be able to restore the previous output handle. Anything short of that is unacceptable for this situation (i.e. I can't restore to CONOUT$, etc.). This needs to have the ability to be recursive. I.e. the following should work as you would expect it to:

std::cout << "A1" << std::endl;

StartStdOutRedirection(TEXT("Foo.txt"));
std::cout << "B1" << std::endl;

StartStdOutRedirection(TEXT("Bar.txt"));
std::cout << "C1" << std::endl;
EndStdOutRedirection();

std::cout << "B2" << std::endl;
EndStdOutRedirection();

std::cout << "A2" << std::endl;

This would be excessively easy if there was a way to "re-sync" stdout as the following code should do the trick:

std::vector<HANDLE> vStdOutHandles;
void StartStdOutRedirection(_In_ LPCTSTR lpFile)
{
    vStdOutHandles.push_back(GetStdHandle(STD_OUTPUT_HANDLE));
    SetStdHandle(STD_OUTPUT_HANDLE, CreateFile(lpFile, GENERIC_WRITE,
        FILE_WRITE_ACCESS | FILE_READ_ACCESS, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL));
}

void EndStdOutRedirection(void)
{
    CloseHandle(GetStdHandle(STD_INPUT_HANDLE));
    SetStdHandle(STD_OUTPUT_HANDLE, vStdOutHandles.back());
    vStdOutHandles.pop_back();
}

The correctness of the above code can be verified by using WriteFile with a call to GetStdHandle(STD_OUTPUT_HANDLE) in place of cout. What I ideally need is an equivalent of freopen that works on HANDLEs. That way I could use DuplicateHandle on the HANDLE returned by GetStdHandle and then this MyReopenHandle function to set the underlying file for that HANDLE to the file of my liking. I believe that would work as I assume both printf and cout have a HANDLE saved somewhere deep down. I tried to "fake it" by duplicating the standard output handle, closing that handle, and then calling CreateFile in hopes that it'll give me the same HANDLE value, but that works sporadically at best. Here's my code for that if you are interested:

std::vector<HANDLE> vStdOutHandles;
bool StartStdOutRedirection(_In_ LPCTSTR lpFile)
{
    bool fResult = false;
    HANDLE hProc = GetCurrentProcess();
    HANDLE hOut = GetStdHandle(STD_OUTPUT_HANDLE);

    if (hOut != INVALID_HANDLE_VALUE)
    {
        HANDLE hDup;
        if (DuplicateHandle(hProc, hOut, hProc, &hDup, 0, FALSE, DUPLICATE_SAME_ACCESS))
        {
            // Need to close the current handle before we open the new one
            CloseHandle(hOut);
            HANDLE hFile = CreateFile(lpFile, GENERIC_WRITE, FILE_WRITE_ACCESS | FILE_READ_ACCESS,
                NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);

            if (hFile != INVALID_HANDLE_VALUE)
            {
                // Should be same HANDLE; else we're screwed...
                assert(hFile == hOut);
                SetStdHandle(STD_OUTPUT_HANDLE, hFile);

                vStdOutHandles.push_back(hDup);
                fResult = true;
            }
            else
            {
                // Otherwise, reopen the previous output HANDLE on failure
                DuplicateHandle(hProc, hDup, hProc, &hFile, 0, FALSE, DUPLICATE_SAME_ACCESS);

                assert(hFile == hOut);
                CloseHandle(hDup);
            }
        }
    }

    return fResult;
}

bool EndStdOutRedirection(void)
{
    bool fResult = false;
    HANDLE hProc = GetCurrentProcess();
    HANDLE hOut = GetStdHandle(STD_OUTPUT_HANDLE);

    if (hOut != INVALID_HANDLE_VALUE && vStdOutHandles.size() != 0)
    {
        HANDLE hDup;
        HANDLE hNext = vStdOutHandles.back();

        // Close current handle and re-open previous one
        CloseHandle(hOut);
        if (DuplicateHandle(hProc, hNext, hProc, &hDup, 0, FALSE, DUPLICATE_SAME_ACCESS))
        {
            // Again, we're screwed if these are not the same
            assert(hOut == hDup);
            SetStdHandle(STD_OUTPUT_HANDLE, hDup);

            vStdOutHandles.pop_back();
            fResult = true;
        }
    }

    return fResult;
}

The above assert fails about half the time (I wasn't really expecting or counting on that to work... I was just interested). That is about as far as I have gotten in terms of this problem. If anyone has any suggestions, please let me know :)

Answer 1

Wow, after a while of searching for a way to manually set the HANDLE of a FILE, I finally discovered that there is a fairly straightforward way to do this using the C Run-Time Library:

std::vector<int> vfdStdOut;
void StartStdOutRedirection(_In_ LPCSTR lpFile)
{
    // Duplicate stdout and give it a new file descriptor
    int fdDup = _dup(_fileno(stdout));
    vfdStdOut.push_back(fdDup);

    // Re-open stdout to the new file
    freopen(lpFile, "w", stdout);
}

bool EndStdOutRedirection(void)
{
    if (vfdStdOut.size() != 0)
    {
        // Get last saved file descriptor and restore it
        int fdNext = vfdStdOut.back();
        _dup2(fdNext, _fileno(stdout));

        // Need to close the file associated with the saved file descriptor
        _close(fdNext);

        vfdStdOut.pop_back();
        return true;
    }

    return false;
}

This will also take care of calling SetStdHandle for you!

Answer 2

Here's a solution I put together (far from perfect of course). It calls a custom function for every character written to STDOUT. In my example, it forwards the stream to OutputDebugString calls.

#include <windows.h>
#include <io.h>
#include <functional>
#include <iostream>

#define STDOUT_FILENO 1
#define STDERR_FILENO 2

enum StdHandleToRedirect {
    STDOUT, STDERR
};

class StdRedirect {
public:
    /// Assumes the specified handle is still assigned to the default FILENO (STDOUT_FILENO/STDERR_FILENO)
    /// TODO allow redirection in every case
    /// callback will run in a new thread and will be notified of any character input to
    /// the specified std handle
    StdRedirect(StdHandleToRedirect h, std::function<void(char)> callback) : callback(callback) {
        CreatePipe(&readablePipeEnd, &writablePipeEnd, 0, 0);
        SetStdHandle(h == STDOUT ? STD_OUTPUT_HANDLE : STD_ERROR_HANDLE, writablePipeEnd);

        // Redirect (TODO: ERROR CHECKING)
        int writablePipeEndFileStream = _open_osfhandle((long)writablePipeEnd, 0);
        FILE* writablePipeEndFile = NULL;
        writablePipeEndFile = _fdopen(writablePipeEndFileStream, "wt");
        _dup2(_fileno(writablePipeEndFile), h == STDOUT ? STDOUT_FILENO : STDERR_FILENO);

        CreateThread(0, 0, (LPTHREAD_START_ROUTINE)stdreader, this, 0, 0);
    }

    // TODO implement destructor, cleanup, reset

private:
    // DWORD (WINAPI *PTHREAD_START_ROUTINE)(LPVOID lpThreadParameter)
    static void WINAPI stdreader(StdRedirect* redirector) {
        while (1) {
            char c;
            DWORD read;
            ::fflush(NULL); // force current stdout to become readable
            // TODO add error handling
            ReadFile(redirector->readablePipeEnd, (void*)&c, 1, &read, 0); // this blocks until input is available
            if (read == 1)
                redirector->callback(c);
        }
    }

    HANDLE readablePipeEnd, writablePipeEnd;
    const std::function<void(char)> callback;
};

int main() {
    std::function<void(char)> toOutputDebugString = [](char x) {
        char str[2] = {x, 0};
        OutputDebugStringA(str);
    };

    StdRedirect so(STDOUT, toOutputDebugString);
    std::cout << "test stdout\n";
    while (1); // busy loop to give the thread time to read stdout. 
    // You might want to look at "Output: Show output from: Debug" now.
    return 0;
}