Powershell piping causes explosive memory usage

Question

I am currently writing a script in Powershell that allows copying a folder in a SVN repository to another while preserving the history. An example of such command is:

svnadmin.exe dump $FromRepoPath `
    | svndumpfilter.exe include --drop-empty-revs --renumber-revs --preserve-revprops $Folder `
    | svnadmin.exe load --ignore-uuid $ToRepoPath

This causes an very high memory usage in Powershell. It appears that Powershell first executes svnadmin.exe and buffers the stdout from SVN admin, then executes svndumpfilter and buffers that output and finally executes svnadmin.exe.

I can work around it by creating a seperate batch file:

@echo off
svnadmin.exe dump %1 | svndumpfilter.exe include --drop-empty-revs --renumber-revs --preserve-revprops %2 | svnadmin.exe load --ignore-uuid %3

And then calling it from Powershell:

cmd.exe /c "SvnHelper.cmd $FromRepoPath $Folder $ToRepoPath"

But this feels like a nasty and unnecessary workaround.

Is there any way to tell Powershell to pass-through directly when piping instead of buffering it?

Answer 1

It's not the output being buffered, but rather the input to any external process. You can verify the behavior with a function like this:

function Read-Pipeline {
[cmdletbinding()]
param ([Parameter(Mandatory = $true, ValueFromPipeline=$true)] $inp)
    Begin {}
    Process {Write-Verbose $inp ; Return $inp}
    End {}
}

If you then run:

.\LongRunning.exe | Read-Pipeline -Verbose | .\Other.exe

You'll see the LongRunning.exe's output in Verbose but Other.exe won't be run until its pipeline is closed. If you do this:

.\LongRunning.exe | Read-Pipeline -Verbose | Write-Host

You'll see alternating Verbose / Console output lines with no buffering because inputs aren't crossing process boundaries.

None of that really helps you. You could work around this by falling back to .NET to launch the processes and manually copying STDOUT to STDIN [1], but it's a lot of work for little reward. The simplest thing to do would be pass your commands to CMD, something like:

& cmd.exe "/C svnadmin.exe dump $FromRepoPath ^| svndumpfilter.exe include --drop-empty-revs --renumber-revs --preserve-revprops $Folder ^| svnadmin.exe load --ignore-uuid $ToRepoPath

[1] http://sushihangover.blogspot.com/2012/01/powershell-piping-standard-output-to.html