Reading other process console output

Question

I have problem when console output is returned in 'waves'. For example console outputs something every second, and for example event triggers 60 times every minute (all events at he same time).

My code:

Process Proc = new Process();
Proc.StartInfo.FileName = SSMS.BinaryDir + "HldsUpdateTool.exe";
Proc.StartInfo.Arguments = "-command update -game tf -dir " + SSMS.RootDir + Key;
Proc.StartInfo.UseShellExecute = false;
Proc.StartInfo.RedirectStandardOutput = true;
Proc.StartInfo.RedirectStandardError = true;

Proc.EnableRaisingEvents = true;
Proc.StartInfo.CreateNoWindow = false;

Proc.ErrorDataReceived += new DataReceivedEventHandler(Proc_ErrorDataReceived);
Proc.OutputDataReceived += new DataReceivedEventHandler(Proc_OutputDataReceived);
Proc.Exited += new EventHandler(Proc_Exited);

Proc.Start();
Proc.BeginErrorReadLine();
Proc.BeginOutputReadLine();

I might suspect that there is problem with update tool. Other programs with console output work fine.

In time line when events are triggered: ( = nothing happened; | event fired )

Should be: ==|==|==|==|==|==|==  
Is: ========|||||||=========||||||=====

Answer 1

You are seeing the effect of the stdout output buffer that the program is using. This is a standard feature of the C runtime library, buffering is enabled when it detects that it is writing to a pipe instead of the console. Instead of automatically flushing after each printf() statement in the program. The buffer is usually around 2 kilobytes. And only gets flushed when it fills up. This greatly enhances efficiency, flushing every time adds a lot of overhead. Important in normal redirect scenarios, when output is written to a file or device.

You can see where this is going, the clumps you see are the content of that buffer. There is no simple fix for this, surgery is required in the program to disable the buffer or flush it where it matters. That's invariably where the buck stops, you would not be doing this if you could alter the program. You could drip-feed your console by buffering what you get from OutputDataReceived but that's a bit silly perhaps.

You won't see this effect when the program is sending output faster than you can process it. Which is pretty common. It effectively gets throttled, blocking while waiting for the output buffer to empty and quickly filling it back up.

There's one more explanation for this, the rate at which OutputReceived can fire also depends on how many threadpool threads you've got running. If that's more than the number of cpu cores, the tp thread that calls OutputReceived may be delayed by a multiple of 0.5 seconds. You would however see the clumping on all programs you redirect.