How can i use Write-Progress in a Pipeline Function?

Question

I'm trying to write a function in PowerShell that accepts pipeline input. I want to display a progress bar using Write-Progress, that increment for every item in the pipeline.

For example:

function Write-PipelineProgress {
    [Cmdletbinding()]
    Param
    (
        [Parameter(Position=0, Mandatory=$true, ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true)] `
        [object[]] $Input,

        [string] $Activity = "Processing items"       
    )

    Begin { Write-Progress -Activity $Activity -Status "Preparing" }

    Process {
        # how do i determine how much progress we've made?
        $percentComplete = ... ?
        Write-Progress -Activity $Activity -Status "Working" -PercentComplete $percentComplete

        # return current item, so processing can continue
        $_
    }

    End { Write-Progress -Activity $Activity -Status "End" -Completed }
}

Get-ChildItem | Write-PipelineProgress -Activity "Listing files"

How can i determine the progress (% complete)?

Answer 1

You need to know the number of items in the pipeline to track progress.

Powershell 3.0 can give you a count of what is in the pipeline without you having to do any work beyond accessing the .Count property of $Input if the pipeline parameter is correctly declared. It also means you can do away with the begin {} process {} end {} blocks and just have a straightforward function if you want.

Earlier versions don't have the Count property so you first have to iterate through and capture the pipeline to get a count then process again, which is not so efficient, as I'll show later.

Powershell V2 Version:

function Show-ProgressV2{
    param (
        [Parameter(Mandatory=$true, Position=0, ValueFromPipeline=$true)]
        [PSObject[]]$InputObject,
        [string]$Activity = "Processing items"
    )

    Begin {$PipeArray = @()}

    Process {$PipeArray+=$InputObject}

    End {
        [int]$TotItems = ($PipeArray).Count
        [int]$Count = 0

        $PipeArray|foreach {
            $_
            $Count++
            [int]$percentComplete = [int](($Count/$TotItems* 100))
            Write-Progress -Activity "$Activity" -PercentComplete "$percentComplete" -Status ("Working - " + $percentComplete + "%")
            }
        }
}    

Powershell V3 Version:

function Show-ProgressV3{
    [CmdletBinding()]
    param (
        [Parameter(Mandatory=$true, Position=0, ValueFromPipeline=$true)]
        [PSObject[]]$InputObject,
        [string]$Activity = "Processing items"
    )

        [int]$TotItems = $Input.Count
        [int]$Count = 0

        $Input|foreach {
            $_
            $Count++
            [int]$percentComplete = ($Count/$TotItems* 100)
            Write-Progress -Activity $Activity -PercentComplete $percentComplete -Status ("Working - " + $percentComplete + "%")
        }
}

Efficiency
Comparing the two, V3 function is around 5-6x faster, depending in the size of the pipeline.

Consider the following pre-filtered file list, looking for all .jpg files in my home drive, selecting the first 200 files and sorting and listing, using the function 3 times in the pipe, per your comment:

$V2 = Measure-Command {Get-ChildItem -Filter *.jpg -Recurse `
| Show-ProgressV2 -Activity "Selecting" `
| Select-Object -First 200 `
| Show-ProgressV2 -Activity "Sorting" `
| Sort-Object -Property FullName `
| Show-ProgressV2 -Activity "Listing" `
| FL}

$V3 = Measure-Command {Get-ChildItem -filter *.jpg -Recurse `
| Show-ProgressV3 -Activity "Selecting" `
| Select-Object -First 200 `
| Show-ProgressV3 -Activity "Sorting" `
| Sort-Object -Property FullName `
| Show-ProgressV3 -Activity "Listing" `
| FL}

$V2  
$V3

Which gives me the the following timings:

PS C:\Users\Graham> C:\Users\Graham\Documents\Stack_ShowProgress_Pipeline.ps1


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 48
Milliseconds      : 360
Ticks             : 483607111
TotalDays         : 0.000559730452546296
TotalHours        : 0.0134335308611111
TotalMinutes      : 0.806011851666667
TotalSeconds      : 48.3607111
TotalMilliseconds : 48360.7111

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 8
Milliseconds      : 335
Ticks             : 83358374
TotalDays         : 9.6479599537037E-05
TotalHours        : 0.00231551038888889
TotalMinutes      : 0.138930623333333
TotalSeconds      : 8.3358374
TotalMilliseconds : 8335.8374

Answer 2

You have to decide how you will display progress. In this example I arbitrarily decided that the begin block will take 5%, the process block will take 90%, and the end block will take 5%. From there you can display intermediate percentages (from 0% to 5% in the begin block for example) in an arbitrary way (which is what this example does in the begin and end blocks). Or you can base the percentages on the amount of processing the script has actually done (which is what process block does). Note that this example is more or less non-sense (but it will execute and show the progress bar.). Also note that the script, as written, wouldn't work in a pipeline, so begin/process/end blocks aren't really necessary, but they don't hurt.

function progress-example {
  param( [string[]] $files)
  begin {
    # This code is somewhat arbitrarily considering the begin block to 
    # take 5% of the time of this cmdlet. The process block is 
    # considered to take 90% of the time, and the end block the last 5%

    write-progress -Activity "Progress Example" -status Beginning `
     -CurrentOperation "Initializing sorted script table" -PercentComplete 0

    # This could be any kind of initialization code; 
    # The code here is non-sense
    $mySortedScriptTable = @{}
    ls *.ps1 | % { $mySortedScriptTable[$_.name] = cat $_ | sort }
    sleep 2 # this code executes too quickly to see the progress bar so slow it down

    write-progress -Activity "Progress Example" -status Beginning `
      -CurrentOperation "Initializing script size table" -PercentComplete 3

    $mySortedScriptSizeTable = @{}
    ls *.ps1 | % { $mySortedScriptSizeTable[$_.name] = $_.Length }

    $totalSizeRequested = 0
    foreach ($file in $files) {
      $totalSizeRequested += $mySortedScriptSizeTable[$file]
    }    
    $numberCharsProcessed = 0
    sleep 2 # this code executes too quickly to see the progress bar so slow it down
    write-progress -Activity "Progress Example" -status Beginning `
      -CurrentOperation "Initialization complete" -PercentComplete 5
    sleep 2 # this code executes too quickly to see the progress bar so slow it down
  }

  process {
    foreach ($file in $files) {
      $thisFileSize = $mySortedScriptSizeTable[$file]

      # Process block takes 90% of the time to complete (90% is arbitrary)
      $percentProcess = 90 * ($numberCharsProcessed / $totalSizeRequested)
      write-progress -Activity "Progress Example" -status "Processing requested files" `
        -CurrentOperation "Starting $file" -PercentComplete (5 + $percentProcess)

      "File $file requested. Size = $thisFileSize. Sorted content follows"
      $mySortedScriptTable[$file]

      $numberCharsProcessed += $thisFileSize
      $percentProcess = 90 * ($numberCharsProcessed / $totalSizeRequested)
      write-progress -Activity "Progress Example" -status "Processing requested files" `
        -CurrentOperation "Starting sleep after $file" -PercentComplete (5 + $percentProcess)

      sleep 2 # slow it down for purposes of demo
    }
  }

  end {
    write-progress -Activity "Progress Example" -status "Final processing" `
      -CurrentOperation "Calculating the fun we had with this example" -PercentComplete (95)
    sleep 2
    write-progress -Activity "Progress Example" -status "Final processing" `
      -CurrentOperation "It's looking like we had mucho fun" -PercentComplete (97)
    sleep 2
    write-progress -Activity "Progress Example" -status "Final processing" `
      -CurrentOperation "Yes, a lot of fun was had" -PercentComplete (99)
    sleep 1
  }
}

progress script1.ps1, script2.ps1, script3.ps1

Answer 3

I publish the Write-ProgressEx cmdlet to avoid routine code. Try it.

Project https://github.com/mazzy-ax/Write-ProgressEx

---

I want to display a progress bar using Write-Progress, that increment for every item in the pipeline.

Use Write-ProgressEx switch -Increment.

The cmdLet store Total and Current. It automatically calculate PercentComplete and SecondsRemaining.

How can i determine the progress (% complete)?

1. The cmdlet Write-ProgressEx calcaulte -PercentComplete automatically if developer provide Total and Current. The cmdlet increment Current if switch -Increment specified.

2. Developer can use Get-ProgressEx to get actual values of parameters.

See examples https://github.com/mazzy-ax/Write-ProgressEx/tree/master/samples

---

More details:

Write-ProgressEx extend the functionality of the standard powershell cmdlet. It provide a simple way to use -PercentComplete and -SecondsRemaining switches.

The cmdlet:

• works with pipe;
• works with empty activity string;
• automatically displays totals;
• automatically calculate percents;
• uses [system.diagnostic.stopwatch] to calculate remaning seconds;
• completes all inner progresses if no parameters;
• stores totals, current values and actual parameters into the module hashtable;
• provide get/set cmdlets to access actual parameters

Note: the cmdlet is not safe with multi-thread.