Parse output from a process that updates a single console line

Question

Greetings stackoverflow members,

in a BackgroundWorker of a WPF Frontend i run sox (open source console sound processing tool) in a System.Diagnostics.Process. In that same way i use several other command line tools and parse their output to poulate progress bars in my frontend.

This works fine for the other tools but not for Sox since instead of spamming new lines for each progress step, it updates a single line on the console by only using carriage returns (\r) and no line feeds (\n). I tried both asynchronous and synchronous reads on process.StandardError.

Using async process.ErrorDataReceived += (sender, args) => FadeAudioOutputHandler(clip, args); in combination with process.BeginErrorReadLine(); doesn't produce any individual status updates because for some reason the carriage returns do not trigger ReadLine, even though the MSDN docs suggest that it should. The output is spit out in one chunk when the process finishes.

I then tried the following code for synchronous char by char reads on the stream:

char[] c;
var line = new StringBuilder();
while (process.StandardError.Peek() > -1)
{
    c = new char[1];
    process.StandardError.Read(c, 0, c.Length);
    if (c[0] == '\r')
    {
        var percentage = 0;
        var regex = new Regex(@"%\s([^\s]+)");
        var match = regex.Match(line.ToString());
        if (match.Success)
        {
            myProgressObject.ProgressType = ProgressType.FadingAudio
            //... some calculations omitted for brevity
            percentage = (int) Math.Round(result);
        }
        else
        {
            myProgressObject.ProgressType = ProgressType.UndefinedStep;
        }
        _backGroundWorker.ReportProgress(percentage, myProgressObject);
        line.Clear();
    }
    else
    {
        line.Append(c[0]);
    }
}

The above code does not seem to read the stream in realtime but will stall output for a while. Then it spams a small chunk and finally deadlocks half-way through the process.

Any hints towards the right direction would be greatly appreciated!

UPDATE with (sloppy?) solution:

This drove me crazy because nothing i tried on the C# side of things seemed to have any effect on the results. My original implementation, before changing it 15 times and introducing new dependencies, was fine.

The problem is with sox and RedirectStandardError alone. I discovered that after grabbing the sox source code and building my own version. First i removed all output of sox entirely except for the stuff i was really interested in and then changing the output to full lines followed by a newline \n . I assumed that this would fix my issues. Well, it didn't. I do not know enough c++ to actually find out why, but they seem to have tempered with how stdio writes to that stream, how it's buffered or do it in such a special way that the streamreader on the c# side is not flushed until the default 4096 byte buffer is full. I confirmed that by padding each line to at least 4096 byte. So in conclusion all i had to do was to manually flush stderr in sox.c after each fprintf(stderr, ...) call in display_status(...):

fflush(stderr);

Though, I'm not sure this is anywhere close to an elegant solution.

Thanks to Erik Dietrich for his answer which made me look at this from a different angle.

Answer 1

The situation you describe is a known problem - for a solution including source code see http://www.codeproject.com/KB/threads/ReadProcessStdoutStderr.aspx

It solves both problems (deadlock and the problem with \n)...

Answer 2

I've had to deal with a similar issue with a bespoke build tool in visual studio. I found that using a regex and doing the parsing in the same thread as the reading is a problem and the output processing grinds to a halt. I ended up with a standard consumer producer solution where you read lines from the output and stick them onto a Queue. Then have the queue be dequeued and processed on some other thread. I can't offer source code but this site has some fantastic resources: http://www.albahari.com/threading/part2.aspx