How much job output will PowerShell hold if I don't receive it?

Question

I have a long-running PowerShell script workflow that is invoked as a job using the -AsJob parameter. It periodically writes warnings and output messages that would need to be retrieved using the Retrieve-Job cmdlet.

I'm curious where PowerShell stores this job output that has not yet been retrieved and if there is a practical upper limit. If so, would it automatically purge job output in a FIFO manner? Crash?

Answer 1

I don't know if this really answers your question, but I tried a variety of Start-Job commands that just created a range from 1 to a a big number - most gave an out of memory exception but I finally got one to work to completion:

# these all bombed out
Start-Job -ScriptBlock { 1..4000000000 }
Start-Job -ScriptBlock { 1..2000000000 }
Start-Job -ScriptBlock { 1..1000000000 }
# this one finally started evaluating
Start-Job -ScriptBlock { 1..100000000 }

I let it run for a little while and it ate up one full CPU core while it was evaluating the range. It ended using about 5GB of RAM according to the Task Manager and would have used more but started to run out of physical memory at that point. The CPU usage dropped as the OS had to start paging memory like crazy.

As you can see in the screenshot below, it actually spawned another powershell.exe process - so I guess that partially answers your question of where does it store the data while the job is running. It appears there is a practical limit that depends on the resources available. PowerShell will not purge or otherwise lose information in the job buffer (unless the second process crashes?) but it may become problematic to retrieve that much data too. You may want to do a few trial runs and see what the acceptable and practical limits are that can be supported by your environment.

Note: Using a much smaller range, like 1..100000, provides more than ample records to observe the behavior while not completely overwhelming the system. I used the larger numbers to see what the practical limits are.