Using the console in a GUI app in windows, only if its run from a console

Question

My application is a GUI app that has helpful (though optional) information through the terminal (via cout).

In Windows I either have a console appear (by compiling as a console app, or allocating it dynamically) or I don't.

My intention is to make use of the console IF it is being run from the console, but ignore the console completely if it was not. (Essentially what happens in Linux and OS X).

I do not wish to redirect to a file (and in the case of using cin, this is not a viable solution anyway).

Is there a way to attach a GUI app in Windows to the console it is run from, if and only if it is run from a console?

Answer 1

and in the case of using cin, this is not a viable solution anyway

This is the killer detail in your question. It is simple on paper, just first call AttachConsole(ATTACH_PARENT_PROCESS) to try to attach to an existing console. That will fail when your program got started from a GUI program like Explorer or a desktop shortcut. So if it returns FALSE then call AllocConsole() to create your own console.

Using cin is a problem however. The command processor pays attention to your EXE and checks if it is console mode app or a GUI app. It will detect a GUI app in your case and then doesn't wait for the process to complete. It displays the prompt again and waits for input. You will then also wait for input but you'll lose, the command processor got there first. Your output is also intermingled with the command prompt, the easy problem to solve.

There's a simple workaround for that, your user should start your program with start /wait yourapp to tell the command processor to wait for the process to complete. Problem is: nobody ever uses that. And the user will not realize what happens when they type input, intending it to go into your program but it is actually interpreted by the command processor. Producing a mystifying error message or formatting the hard drive.

Only two good ways to solve this unsolvable problem. Either build your program as a console mode app and call FreeConsole() when you find out you want to display a GUI. Or always call AllocConsole(). These are not great alternatives. The first approach is the one used by the Java JVM on Windows. One of the oldest bugs filed against the JVM and driving Java programmers completely batty from the flashing console window.

The third alternative is the only decent one, and the one you don't want, create another EXE that will always use the console. Like Java does, javaw.exe vs java.exe.

A trick is possible, you can rename that file from "yourapp2.exe" to "yourapp.com". It will be picked first when the user types "yourapp" at the command line prompt, a desktop shortcut can still point to "yourapp.exe". Visual Studio uses this trick, devenv.com vs devenv.exe.

Answer 2

You can check CONSOLE_SCREEN_BUFFER_INFO (via GetConsoleScreenBufferInfo) on startup to determine if you've been run from within an existing console. If the buffer's position is 0,0, you were run from outside of the console. For details, see this Microsoft Knowledgebase Article which describes the process.

In order for this to work, you need to compile your application as a console application (using /SUBSYSTEM:CONSOLE), and then detach yourself from the console if the application started a new console (buffer at 0,0). This will cause the program to properly "attach" to the calling console when launched from a command line.