FreeConsole behaviour on Windows 8

Question

On Windows 8, we have an issue with FreeConsole. It seems to close the stdio handles, without shutting the file streams.

This may be a Windows 8 problem, or it could be that I simply don't understand the (totally absurd) way the Windows console/GUI app subsystem does things.

What's going on?

Minimal example below. Tested with compilers: VS2005, VS2013, VS2017, using statically linked CRT.

#include <windows.h>
#include <io.h>
#include <stdio.h>

static void testHandle(FILE* file) {
  HANDLE h = (HANDLE)_get_osfhandle(fileno(file));
  DWORD flags;
  if (!GetHandleInformation(h, &flags)) {
    MessageBoxA(0, "Bogus handle!!", "TITLE", MB_OK);
  }
}

int main(int argc, char** argv)
{
  freopen("NUL", "wb", stdout); // Demonstrate the issue with NUL
  // Leave stderr as it is, to demonstrate the issue with handles
  // to the console device.

  FreeConsole();

  testHandle(stdout);
  testHandle(stderr);
}

Answer 1

issue caused by fact that prior Windows 8 standard (not redirected) console handles (that returned by GetStdHandle) where actually pseudohandles, wich values doesnt instersect with other kernel object handles, so writing to that pseudohandle after it being 'closed' by FreeConsole always fails. In Win8 MS changed something inside, so GetStdHandle returns normal kernel object handle that refers to console subsystem driver object (actually that driver also appeared only in Win8). So FreeConsole closes that handle. The most funny thing that CRT does GetStdHandle on startup and saves returned value somewhere inside and uses wherever use called C functions that access std::in/out/err. Since FreeConsole closed that handle, and its not a special pseudohandle value anymore - same handle value can be reused by any other opened kernel object handle, and you will be lucky if it will not be file, pipe, or socket cause in this case all your debug outpout will go there:)

Answer 2

After disassembling the code for FreeConsole on different Windows versions, I worked out the cause of the problem.

FreeConsole is a remarkably unsubtle function! I really does close a load of handles for you, even if it doesn't "own" those handles (eg HANDLEs owned by stdio functions).

And, the behaviour is different in Windows 7 and 8, and changed again in 10.

Here's the dilemma when coming up with a fix:

• Once stdio has a HANDLE to the console device, there is no documented way to get it to give up that handle, without a call to CloseHandle. You can call close(1) or freopen(stdout) or whatever you like, but if there is an open file descriptor that refers to the console, CloseHandle will be called on it, if you want to switch stdout to a new NUL handle after FreeConsole.
• On the other hand, since Windows 10 there's also no way to avoid FreeConsole calling CloseHandle as well.
• Visual Studio's debugger and the Application Verifier flag the app for calling CloseHandle on an invalid HANDLE. And, they're right, it's really not good.
• So, if you try and "fix up" stdio before the call to FreeConsole then FreeConsole will do an invalid CloseHandle (using its cached handle, and there's no way whatsoever to tell it that handle's gone - FreeConsole no longer checks GetStdHandle(STD_OUTPUT_HANDLE)). And, if you call FreeConsole first, there's no way to fix up the stdio objects without causing them to do an invalid call to CloseHandle.

By elimination, I conclude that the only solution is to use an undocumented function, if the public ones just won't work.

// The undocumented bit!
extern "C" int __cdecl _free_osfhnd(int const fh);
static HANDLE closeFdButNotHandle(int fd) {
  HANDLE h = (HANDLE)_get_osfhandle(fd);
  _free_osfhnd(fd); // Prevent CloseHandle happening in close()
  close(fd);
  return h;
}

static bool valid(HANDLE h) {
  SetLastError(0);
  return GetFileType(h) != FILE_TYPE_UNKNOWN || GetLastError() == 0;
}

static void openNull(int fd, DWORD flags) {
  int newFd;
  // Yet another Microsoft bug! (I've reported four in this code...)
  // They have confirmed a bug in dup2 in Visual Studio 2013, fixed
  // in Visual Studio 2017.  If dup2 is called with fd == newFd, the
  // CRT lock is corrupted, hence the check here before calling dup2.
  if (!_tsopen_s(&newFd, _T("NUL"), flags, _SH_DENYNO, 0) &&
      fd != newFd)
    dup2(newFd, fd);
  if (fd != newFd) close(newFd);
}

void doFreeConsole() {
  // stderr, stdin are similar - left to the reader.  You probably
  // also want to add code (as we have) to detect when the handle
  // is FILE_TYPE_DISK/FILE_TYPE_PIPE and leave the stdio FILE
  // alone if it's actually pointing to disk/pipe.
  HANDLE stdoutHandle = closeFdButNotHandle(fileno(stdout)); 

  FreeConsole(); // error checking left to the reader

  // If FreeConsole *didn't* close the handle then do so now.
  // Has a race condition, but all of this code does so hey.
  if (valid(stdoutHandle)) CloseHandle(stdoutHandle);

  openNull(stdoutRestore, _O_BINARY | _O_RDONLY);
}