PowerShell: the mysterious -RemainingScripts parameter of ForEach-Object

Question

Short question: anyone has detailed information on -RemainingScripts parameter of ForEach-Object?

Long question:

I just started learning PowerShell from last week and I'm going through each Cmdlets to learn more details. Based on public documentation, we know ForEach-Object can have Begin-Process-End blocks, like this:

Get-ChildItem | foreach -Begin { "block1";
    $fileCount = $directoryCount = 0} -Process { "block2";
    if ($_.PsIsContainer) {$directoryCount++} else {$fileCount++}} -End {
    "block3"; "$directoryCount directories and $fileCount files"}

Result is expected: 1 time for "block1" and "block3", "block2" is repeated for each item passed in, and dir count/file count are all correct. So far so good.

Now, what's interesting is, the following command also works and gives exactly the same result:

Get-ChildItem | foreach { "block1"
    $fileCount = $directoryCount = 0}{ "block2";
    if ($_.PsIsContainer) {$directoryCount++} else {$fileCount++}}{
    "block3"; "$directoryCount directories and $fileCount files"}

Just 3 ScriptBlocks passed to foreach. Based on manual, the first one goes to -Process (Position 1). But how about the remaining 2? According to manual, there's not a parameter with "Position 2". So I turned to Trace-Command, and found that the later 2 script blocks were actually to RemainingScripts as "IList with 2 elements".

BIND arg [$fileCount = $directoryCount = 0] to parameter [Process]
BIND arg [System.Management.Automation.ScriptBlock[]] to param [Process] SUCCESSFUL
BIND arg [System.Collections.ArrayList] to parameter [RemainingScripts]
BIND arg [System.Management.Automation.ScriptBlock[]] to param [RemainingScripts] SUCCESSFUL

So If I change the command to this:

# No difference with/without the comma "," between the last 2 blocks
Get-ChildItem | foreach -Process { "block1"
    $fileCount = $directoryCount = 0} -RemainingScripts { "block2";
    if ($_.PsIsContainer) {$directoryCount++} else {$fileCount++}},{
    "block3"; "$directoryCount directories and $fileCount files"}

Still, exactly the same result.

So as you noticed, all 3 commands give the same result. This raises the interesting question: both of the later two commands (implicitly) specified -Process, however ForEach-Object surprisingly ends up using the argument of -Process as "-Begin"! (script block is executed once at the beginning).

This experimentation suggests:

1. -RemainingScripts parameter will take all unbound ScriptBlocks
2. When 3 blocks are passed in, although the first one goes to -Process, later it is actually used as "Begin" while the remaining 2 become "Process" and "End"

Still, all above are just my wild guess. I didn't find documentation to support my guess

So, finally we go back to my short question :) Anyone has detailed information on -RemainingScripts parameter of ForEach-Object?

Thanks.

Answer 1

I did more research and now feel confident to answer the behavior of -RemainingScripts parameter when multiple ScriptBlocks are passed in.

If you run the following commands and inspect the result carefully, you will find the pattern. It's not quite straightforward, but still not hard to figure out.

1..5 | foreach { "process block" } { "remain block" }
1..5 | foreach { "remain block" }  -Process { "process block" }
1..5 | foreach { "remain block" } -End { "end block" } -Process { "process block" } -Begin { "begin block" }
1..5 | foreach { "remain block 1" } -End { "end block" } -Process { "process block" } { "remain block 2" }
1..5 | foreach { "remain block 1" } { "remain block 2" } -Process { "process block" } -Begin { "begin block" }
1..5 | foreach { "remain block 1" } { "remain block 2" } -Process { "process block" } { "remain block 3" }
1..5 | foreach { "process block" } { "remain block 1" } { "remain block 2" } -Begin { "begin block" }
1..5 | foreach { "process block" } { "remain block 1" } { "remain block 2" } { "remain block 3" }

So what's the pattern here?

• When there's single ScriptBlock passed in: easy, it just goes to -Process (the most common usage)

• When exactly 2 ScriptBlocks are passed in, there are 3 possible combinations

  1. -Process & -Begin -> execute as specified
  2. -Process & -End -> execute as specified
  3. -Process & -RemainingScripts -> Process becomes Begin, while RemainingScripts becomes Process

If we run these 2 statements:

1..5 | foreach { "process block" } { "remain block" }
1..5 | foreach { "remain block" }  -Process { "process block" }

# Both of them will return:
process block
remain block
remain block
remain block
remain block
remain block

As you will find out, this is just a special case of the following test case:

• When more than 2 ScriptBlocks are passed in, follow this workflow:

  1. Bind all scriptblocks as specified (Begin,Process,End); remaining ScriptBlocks go to RemainingScripts
  2. Order all scripts as: Begin > Process > Remaining > End

  3. Result of ordering is a collection of ScriptBlocks. Let's call this collection OrderedScriptBlocks

     • If Begin/End are not bound, just ignore

  4. (Internally) Re-bind parameters based on OrderedScriptBlocks

     • OrderedScriptBlocks[0] becomes Begin
     • OrderedScriptBlocks[1..-2] become Process
     • OrderedScriptBlocks[-1] (the last one) becomes End

Let's take this example

1..5 | foreach { "remain block 1" } { "remain block 2" } -Process { "process block" } { "remain block 3" }

Order result is:

{ "process block" }    # new Begin
{ "remain block 1" }   # new Process
{ "remain block 2" }   # new Process
{ "remain block 3" }   # new End

Now the execution result is completely predictable:

process block
remain block 1
remain block 2
remain block 1
remain block 2
remain block 1
remain block 2
remain block 1
remain block 2
remain block 1
remain block 2
remain block 3

That's the secret behind -RemainingScripts and now we understand more internal behavior of ForEach-Object!

Still I have to admit there's no documentation to support my guess (not my fault!), but these test cases should be enough to explain the behavior I described.

Answer 2

I believe -remainingscripts (with attribute 'ValueFromRemainingArguments') is to enable an idiom like this from Windows Powershell in Action, an idiom which almost nobody knows (20% of Powershell is only documented in that book):

Get-ChildItem | ForEach {$sum=0} {$sum++} {$sum}

The blocks end up acting like begin process end. The parameters being used are actually -process and -remainingscripts.

trace-command -name parameterbinding { Get-ChildItem | ForEach-Object {$sum=0} {$sum++} {$sum} } -PSHost

That trace-command seems to confirm this.

Here's a simple demo of ValueFromRemainingArguments with scriptblocks.

function remaindemo {
  param ($arg1,  [Parameter(ValueFromRemainingArguments)]$remain)
  & $arg1
  foreach ($i in $remain) {
    & $i
  }
}

remaindemo { 'hi' } { 'how are you' } { 'I am fine' }

Other commands with ValueFromRemainingArguments parameters:

gcm -pv cmd | select -exp parametersets | select -exp parameters |
  where ValueFromRemainingArguments | 
  select @{n='Cmdname';e={$cmd.name}},name

Cmdname        Name
-------        ----
ForEach-Object RemainingScripts
ForEach-Object ArgumentList
Get-Command    ArgumentList
Get-Command    ArgumentList
Join-Path      AdditionalChildPath
New-Module     ArgumentList
New-Module     ArgumentList
Read-Host      Prompt
Trace-Command  ArgumentList
Write-Host     Object
Write-Output   InputObject

Answer 3

Here are some more examples that support @FangZhou's ordering hypothesis.

If you specify the other blocks, it seems to make sense how it works:

PS C:\> 1..5 | ForEach-Object -Begin { "Begin: $_" } -End { "End: $_" } -Process { "Process: $_" } -RemainingScripts { "R1: $_" },{ "R2: $_" },{ "R3: $_" }
Begin: 
Process: 1
R1: 1
R2: 1
R3: 1
Process: 2
R1: 2
R2: 2
R3: 2
Process: 3
R1: 3
R2: 3
R3: 3
Process: 4
R1: 4
R2: 4
R3: 4
Process: 5
R1: 5
R2: 5
R3: 5
End: 

Even if you pass empty blocks:

PS C:\> 1..5 | ForEach-Object -Begin {} -End {} -Process { "Process: $_" } -RemainingScripts { "R1: $_" },{ "R2: $_" },{ "R3: $_" }
Process: 1
R1: 1
R2: 1
R3: 1
Process: 2
R1: 2
R2: 2
R3: 2
Process: 3
R1: 3
R2: 3
R3: 3
Process: 4
R1: 4
R2: 4
R3: 4
Process: 5
R1: 5
R2: 5
R3: 5

However, if you don't specify -End, it does something completely different. The last scriptblock passed to the command is used for -End.

PS C:\> 1..5 | ForEach-Object -Begin { "Begin: $_" } -Process { "Process: $_" } -RemainingScripts { "R1: $_" },{ "R2: $_" },{ "R3: $_" }
Begin: 
Process: 1
R1: 1
R2: 1
Process: 2
R1: 2
R2: 2
Process: 3
R1: 3
R2: 3
Process: 4
R1: 4
R2: 4
Process: 5
R1: 5
R2: 5
R3: 

And you can change what happens by changing the order of properties:

PS C:\> 1..5 | ForEach-Object  -RemainingScripts { "R1: $_" },{ "R2: $_" },{ "R3: $_" } -Begin { "Begin: $_" } -Process { "Process: $_" }
Begin: 
R1: 1
R2: 1
R3: 1
R1: 2
R2: 2
R3: 2
R1: 3
R2: 3
R3: 3
R1: 4
R2: 4
R3: 4
R1: 5
R2: 5
R3: 5
Process: 

And if you don't specify -Begin, it's different again. Now the first scriptblock passed is used for -Begin:

PS C:\> 1..5 | ForEach-Object -End { "End: $_" } -Process { "Process: $_" } -RemainingScripts { "R1: $_" },{ "R2: $_" },{ "R3: $_" }
Process: 
R1: 1
R2: 1
R3: 1
R1: 2
R2: 2
R3: 2
R1: 3
R2: 3
R3: 3
R1: 4
R2: 4
R3: 4
R1: 5
R2: 5
R3: 5
End: 

If you specify neither -Begin nor -End, it combines the two. Now the first scriptblock replaces -Begin and the last scriptblock of replaces -End:

PS C:\> 1..5 | ForEach-Object -Process { "Process: $_" } -RemainingScripts { "R1: $_" },{ "R2: $_" },{ "R3: $_" }
Process: 
R1: 1
R2: 1
R1: 2
R2: 2
R1: 3
R2: 3
R1: 4
R2: 4
R1: 5
R2: 5
R3: 

As far as I can tell, it's intended to support positional scriptblocks, where you're going to write:

1..5 | ForEach-Object { "Begin: $_" } { "Process1: $_" } { "Process2: $_" } { "Process3: $_" } { "End: $_" }

Or like so:

1..5 | ForEach-Object { "Begin: $_" },{ "Process1: $_" },{ "Process2: $_" },{ "Process3: $_" },{ "End: $_" }

Both of which output:

Begin: 
Process1: 1
Process2: 1
Process3: 1
Process1: 2
Process2: 2
Process3: 2
Process1: 3
Process2: 3
Process3: 3
Process1: 4
Process2: 4
Process3: 4
Process1: 5
Process2: 5
Process3: 5
End: