Using stdout in a Win32 GUI application: crashes if I don't have a redirect to file in arguments

Question

I'm building a Win32 GUI app. Inside that app, I'm using a DLL that was intended to be used in a command line app.

Suppose Foo.exe is my GUI app, and bar() is a function in the DLL that prints "hello" to stdout. Foo.exe calls bar().

If I run Foo.exe from the command line with a redirect (>) (i.e. Foo.exe > out.txt) it writes "hello" to out.txt and exits normally (as expected).

However, if I run Foo.exe without a redirect (either from cmd.exe or by double-clicking in Windows Explorer), it crashes when bar() is called.

If I run Foo.exe inside the debugger with the redirect in the command line (set through VS's properties for the project) and call "GetStdHandle(STD_OUTPUT_HANDLE)", I get a reasonable address for a handle. If I call it without the redirect in the command line, I get 0.

Do I need something to "initialize" standard out? Is there a way that I can set up this redirect in the application startup? (Redirecting to a file would be ideal. But just throwing out the data printed by the DLL would be okay, too.)

Finally, I suspect that the DLL is writing to stdout through the CRT POSIX-like API, because it is a cross-platform DLL. I don't know if this matters.

I've tried creating a file with CreateFile and calling SetStdHandle, but that doesn't seem to work. I may be creating the file incorrectly, however. See code below.

HANDLE hStdOut = GetStdHandle(STD_OUTPUT_HANDLE);
// hStdOut is zero  

HANDLE hFile;
hFile = CreateFile(TEXT("something.txt"),        // name of the write
                   GENERIC_WRITE,               // open for writing
                   0,                           // do not share
                   NULL,                        // default security
                   CREATE_NEW,             // create new file only
                   FILE_ATTRIBUTE_NORMAL,  // normal file
                   NULL);                  // no attr. template
BOOL r = SetStdHandle(STD_OUTPUT_HANDLE, hFile) ;   
hStdOut = GetStdHandle(STD_OUTPUT_HANDLE);
// hStdOut is now equal to hFile, and r is 1

bar();
// crashes if there isn't a redirect in the program arguments

UPDATE: I just found this article: http://support.microsoft.com/kb/105305. It states "Note that this code does not correct problems with handles 0, 1, and 2. In fact, due to other complications, it is not possible to correct this, and therefore it is necessary to use stream I/O instead of low-level I/O."

My DLL definitely uses file handles 0,1 and 2. So, there may be no good solution to this problem.

I'm working on a solution that checks for this case, and re-launches the exe appropriately using CreateProcess. I'll post here when I'm done.

Answer 1

The solution that I've found is the following:

• obtain a valid File HANDLE in some way to direct the standard output. Lets call the file handle "fh". (please note that on Windows a File HANDLE is not the same thing of a file descriptor)
• associate a file descriptor to the file handle with _open_osfhandle (see http://msdn.microsoft.com/en-us/library/kdfaxaay.aspx for details) Lets call "fd" the new file descriptor, an int value.
• call dup2 to associate the STDOUT_FILENO to the given file descriptor: dup2(fd, STDOUT_FILENO)
• create a file strem associated to the stdout file descriptor FILE* f = _fdopen(STDOUT_FILENO, "w");
• memset stdout to the content of f: *stdout = *f
• call SetStdHandle on the given file handle: SetStdHandle(STD_OUTPUT_HANDLE, ofh);

Please note that I've not tested exactly this sequence but something slightly different.

I don't know if some steps are redundant.

In any case the following article explain very well the concept of file handle, descriptor et fiel stream:

http://dslweb.nwnexus.com/~ast/dload/guicon.htm