Output to command-line if started from command line

Question

I'm writing an application that can be started either as a standard WinForms app or in unattended mode from the command-line. The application was built using the VS 2k5 standard WinForms template.

When the application is executed from the command-line, I want it to output information that can be captured by the script executing the application. When I do this directly from Console.WriteLine(), the output does not appear, although it can be captured by piping to a file.

On the other hand, I can force the application to pop up a second console by doing a P/Invoke on AllocConsole() from kernel32. This is not what I want, though. I want the output to appear in the same window the application was called from.

This is the salient code that allows me to pop up a console from the command line:

<STAThread()> Public Shared Sub Main()

    If My.Application.CommandLineArgs.Count = 0 Then
        Dim frm As New ISECMMParamUtilForm()
        frm.ShowDialog()
    Else
        Try
            ConsoleControl.AllocConsole()
            Dim exMan As New UnattendedExecutionManager(ConvertArgs())
            IsInConsoleMode = True
            OutputMessage("Application started.")
            If Not exMan.SetSettings() Then
                OutputMessage("Execution failed.")
            End If
        Catch ex As Exception
            Console.WriteLine(ex.ToString())
        Finally
            ConsoleControl.FreeConsole()
        End Try

    End If

End Sub

Public Shared Sub OutputMessage(ByVal msg As String, Optional ByVal isError As Boolean = False)
    Trace.WriteLine(msg)
    If IsInConsoleMode Then
        Console.WriteLine(msg)
    End If

    If isError Then
        EventLog.WriteEntry("ISE CMM Param Util", msg, EventLogEntryType.Error)
    Else
        EventLog.WriteEntry("ISE CMM Param Util", msg, EventLogEntryType.Information)
    End If

End Sub

Answer 1

Raymond Chen recently posted (a month after the question was posted here on SO) a short article about this:

How do I write a program that can be run either as a console or a GUI application?

You can't, but you can try to fake it.

Each PE application contains a field in its header that specifies which subsystem it was designed to run under. You can say IMAGE_SUBSYSTEM_WINDOWS_GUI to mark yourself as a Windows GUI application, or you can say IMAGE_SUBSYSTEM_WINDOWS_CUI to say that you are a console application. If you are GUI application, then the program will run without a console.

The subsystem determines how the kernel prepares the execution environment for the program. If the program is marked as running in the console subsystem, then the kernel will connect the program's console to the console of its parent, creating a new console if the parent doesn't have a console. (This is an incomplete description, but the details aren't relevant to the discussion.) On the other hand, if the program is marked as running as a GUI application, then the kernel will run the program without any console at all.

In that article he points to another by Junfeng Zhang that discusses how a couple of programs (Visual Studio and ildasm) implement this behavior:

How to make an application as both GUI and Console application?

In VisualStudio case, there are actually two binaries: devenv.com and devenv.exe. Devenv.com is a Console app. Devenv.exe is a GUI app. When you type devenv, because of the Win32 probing rule, devenv.com is executed. If there is no input, devenv.com launches devenv.exe, and exits itself. If there are inputs, devenv.com handles them as normal Console app.

In ildasm case, there is only one binary: ildasm.exe. It is first compiled as a GUI application. Later editbin.exe is used to mark it as console subsystem. In its main method it determines if it needs to be run as console mode or GUI mode. If need to run as GUI mode, it relaunches itself as a GUI app.

In the comments to Raymond Chen's article, laonianren has this to add to Junfeng Zhang's brief description of how Visual Studio works:

devenv.com is a general purpose console-mode stub application. When it runs it creates three pipes to redirect the console's stdin, stdout and stderr. It then finds its own name (usually devenv.com), replaces the ".com" with ".exe" and launches the new app (i.e. devenv.exe) using the read end of the stdin pipe and the write ends of the stdout and stderr pipes as the standard handles. Then it just sits and waits for devenv.exe to exit and copies data between the console and the pipes.

Thus even though devenv.exe is a gui app it can read and write the "parent" console using its standard handles.

And you could use devenv.com yourself for myapp.exe by renaming it to myapp.com. But you can't in practise because it belongs to MS.

Answer 2

Update 1:

As said in Michael Burr answer, Raymond Chen recently posted a short article about this. I am happy to see that my guess was not totally wrong.

Update 0:

Disclaimer: This "answer" is mostly speculation. I post it only because enough time has passed to establish that not many people have the answer to what look like a fundamental question.

I think that the "decision" if the application is gui or console is made at compile time and not at runtime. So if you compile your application as gui application, even if you don't display the gui, its still a gui application and doesn't have console. If you choose to compile it as console application then at minimum you will have a console windows flashing before moving to gui "mode". And I don't know if it is possible in managed code.

The problem is fundamental, I think, Because a console application has to take "control" of the calling console application. And it has to do so before the code of the child application is running.