PowerShell stderr redirect to file inserts newlines

Question

Edit: I created a PowerShell UserVoice "suggestion" for (against?) this behavior; feel free to upvote.

PowerShell (5.1.16299.98, Windows 10 Pro 10.0.16299) is inserting newlines into my stderr when I redirect to file—as if to format for the console. Let's generate error messages of arbitrary length:

class Program
{
    static void Main(string[] args)
    {
        System.Console.Error.WriteLine(new string('x', int.Parse(args[0])));
    }
}

I compiled the above to longerr.exe. Then I call it like this:

$ .\longerr.exe 60 2>error.txt

I ran the following script in a PowerShell console with window width 60:

$h = '.\longerr.exe : '.Length
$w = 60 - 1
$f = 'error.txt'
Remove-Item $f -ea Ignore
(($w-$h), ($w-$h+1), ($w), ($w+1), ($w*2-$h), ($w*2-$h+1)) |
    % { 
        $_ >> $f
        .\longerr.exe $_ 2>>$f
    }

Now in a wider console I ran the following:

$ Get-Content $f | Select-String '^(?![+\t]|At line|  )'

(I could have just opened the file in a text editor and trimmed lines.) Here's the output:

43
.\longerr.exe : xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

44
.\longerr.exe : 
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

59
.\longerr.exe : 
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

60
.\longerr.exe : xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxx

102
.\longerr.exe : xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

103
.\longerr.exe : xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
x

Why is PowerShell doing this? Can I make it stop? I'd rather not have to do something like this:

        .\longerr.exe $_ 2>&1 |
            % {
                if ($_ -is [System.Management.Automation.ErrorRecord]) {
                    $_.Exception.Message | Out-File -FilePath $f -Append
                }
            }

First, all the opening and closing of that file can be slow (I know I can add more code and use a StreamWriter), and second, there is still a problem (bug?) with that approach, which I won't go into in this question.

For a sanity check, I ran longerr.exe 1000 2>test.txt in cmd.exe; it inserted no spurious linebreaks.

Answer 1

Thanks to TessellatingHeckler's comment pointing me to the question Why does PowerShell chops message on stderr?, I've been able to fix both problems in my question—the main one and the one I mentioned at the end. The key is the following:

The complete output on stderr of the executable is simply split across several objects of type System.Management.Automation.ErrorRecord. The actual splitting seems to be non deterministic (*). Moreover, the partial strings are stored inside the property Exception instead of TargetObject. Only the first ErrorRecord has a non-null TargetObject.

⋮

(*) It depends on the order of write/flush calls of the program in relation to the read calls of the Powershell. If one adds a fflush(stderr) after each fprintf() in my test program below, there will be much more ErrorRecord objects. Except the first one, which seems deterministic, some of them include 2 output lines and some of them 3.

With this, I was able to modify longerr.exe to also reproduce the bug I alluded to at the end:

class Program
{
    static void Main(string[] args)
    {
        if (args.Length == 1)
        {
            System.Console.Error.WriteLine(new string('x', int.Parse(args[0])));
        }
        else
        {
            for (int i = 0; i < int.Parse(args[1]); i++)
            {
                System.Console.Error.WriteLine("\n");
                System.Console.Error.WriteLine(new string('x', int.Parse(args[0])));
            }
        }
    }
}

Here's the PowerShell script which works (and efficiently):

$p_out = 'success.txt'
$p_err = 'error.txt'

try
{
    [Environment]::CurrentDirectory = $PWD
    $append = $false
    $out = [System.IO.StreamWriter]::new($p_out, $append)
    $err = [System.IO.StreamWriter]::new($p_err, $append)

    .\longerr.exe 2000 4 2>&1 |
        % {
            if ($_ -is [System.Management.Automation.ErrorRecord]) {
                # https://stackoverflow.com/a/33858097/2328341
                if ($_.TargetObject -ne $null) {
                    $err.WriteLine(); 
                }
                $err.Write($_.Exception.Message)
            } else {
                $out.WriteLine($_)
            }
        }
}
finally
{
    $out.Close()
    $err.Close()
}

Notes:

1. Without the test for TargetObject, I would eliminate the main problem I was focusing on in my question, but I would still get the "bug" I mentioned at the end, which the linked SO question addresses.
2. I could have used Out-File (with -NoNewline) instead of StreamWriter, but there are two problems with that:
   1. Windows Defender was making all the file opens and closes over an order of magnitude slower than when I turned off "Real-time protection" (either globally or on the directory containing error.txt and success.txt).
   2. Even without Defender slowing things down, StreamWriter out-performs Out-File by over an order of magnitude. For reference, I'm using a Samsung 960 EVO for storage.
3. The StreamWriter(string, bool) constructor writes UTF-8 with no Byte-Order Mark (BOM), while PowerShell 5.1's redirection operators > and >> use UTF-16 LE with BOM. For reference, PowerShell 6.0 defaults to UTF-8 with no BOM.

(I've included stdout for completeness in real-world situations.) Now, that's an absolutely ridiculous amount of work to get the following functionality of cmd.exe:

$ .\longerr.exe 2000 4 2>error.txt