Wait for multiple simultaneous powershell commands in other sessions to finish before running next commands

Question

I am trying to get a master powershell script to do the following:

1. Run commands in 3 other powershell sessions (they all go for about ~1h - I'd like them to run concurrently, so that the jobs they do can all get done at the same time)
2. Wait for all 3 other powershell sessions to finish
3. Continue on with remaining commands in the initial powershell window

Extremely simple example

My real use case is similar to the following, except the times always vary, ECHO "hi" should happen only once all the other (3) commands have finished (in this case we know they'll take 10000 seconds, but in my actual use case this varies a lot). Also note, it's not clear which of the 3 commands will take the longest each time.

start powershell { TIMEOUT 2000 }
start powershell { TIMEOUT 3000 }
start powershell { TIMEOUT 10000 }
ECHO "hi"

I can see (here) that I can put an & in front of the command in order to tell powershell to wait until it's complete before progressing to subsequent commands. However, I do not know how to do so with 3 simultaneous commands

Answer 1

You are indeed looking for Powershell background jobs, as Lee Daily advises.

However, jobs are heavy-handed, because each job runs in its own process, which introduces significant overhead, and can also result in loss of type fidelity (due to PowerShell's XML-based serialization infrastructure being involved - see this answer).

The ThreadJob module offers a lightweight alternative based on threads. It comes with PowerShell [Core] v6+ and in Windows PowerShell can be installed on demand with, e.g.,
Install-Module ThreadJob -Scope CurrentUser.^[1]

You simply call Start-ThreadJob instead of Start-Job, and use the standard *-Job cmdlets to manage such thread jobs - the same way you'd manage a regular background job.

Here's an example:

$startedAt = [datetime]::UtcNow

# Define the commands to run as [thread] jobs.
$commands = { $n = 2; Start-Sleep $n; "I ran for $n secs." }, 
            { $n = 3; Start-Sleep $n; "I ran for $n secs." }, 
            { $n = 10; Start-Sleep $n; "I ran for $n secs." }

# Start the (thread) jobs.
# You could use `Start-Job` here, but that would be more resource-intensive
# and make the script run considerably longer.
$jobs = $commands | Foreach-Object { Start-ThreadJob $_ }

# Wait until all jobs have completed, passing their output through as it
# is received, and automatically clean up afterwards.
$jobs | Receive-Job -Wait -AutoRemoveJob


"All jobs completed. Total runtime in secs.: $(([datetime]::UtcNow - $startedAt).TotalSeconds)"

The above yields something like the following; note that the individual commands' output is reported as it becomes available, but execution of the calling script doesn't continue until all commands have completed:

I ran for 2 secs.
I ran for 3 secs.
I ran for 10 secs.
All jobs completed. Total runtime in secs.: 10.2504931

Note: In this simple case, it's obvious which output came from which command, but more typically the output from the various jobs will run unpredictably interleaved, which makes it difficult to interpret the output - see the next section for a solution.

As you can see, the overhead introduced for the thread-based parallel execution in the background is minimal - overall execution took only a little longer than 10 seconds, the runtime of the longest-running of the 3 commands.

If you were to use the process-based Start-Job instead, the overall execution time might look something like this, showing the significant overhead introduced, especially the first time you run a background job in a session:

All jobs completed. Total runtime in secs.: 18.7502717

That is, at least on the first invocation in a session, the benefits of parallel execution in the background were negated - execution took longer than sequential execution would have taken in this case.

While subsequent process-based background jobs in the same session run faster, the overhead is still significantly higher than it is for thread-based jobs.

---

Synchronizing the job output streams

If you want show output from the background commands per command, you need to collect output separately.

Note: In a console window (terminal), this requires you to wait until all commands have completed before you can show the output (because there is no way to show multiple output streams simultaneously via in-place updating, at least with the regular output commands).

$startedAt = [datetime]::UtcNow

$commands = { $n = 1; Start-Sleep $n; "I ran for $n secs." }, 
            { $n = 2; Start-Sleep $n; "I ran for $n secs." }, 
            { $n = 3; Start-Sleep $n; "I ran for $n secs." }

$jobs = $commands | Foreach-Object { Start-ThreadJob $_ }

# Wait until all jobs have completed.
$null = Wait-Job $jobs

# Collect the output individually for each job and print it.
foreach ($job in $jobs) {
  "`n--- Output from {$($job.Command)}:"
  Receive-Job $job
} 

"`nAll jobs completed. Total runtime in secs.: $('{0:N2}' -f ([datetime]::UtcNow - $startedAt).TotalSeconds)"

The above will print something like this:

--- Output from { $n = 1; Start-Sleep $n; "I ran for $n secs." }:
I ran for 1 secs.

--- Output from { $n = 2; Start-Sleep $n; "I ran for $n secs." }:
I ran for 2 secs.

--- Output from { $n = 3; Start-Sleep $n; "I ran for $n secs." }:
I ran for 3 secs.

All jobs completed. Total runtime in secs.: 3.09

---

Using Start-Process to run the commands in separate windows

On Windows, you can use Start-Process (whose alias is start) to run commands in a new window, which is also asynchronous by default, i.e., serially launched commands do run in parallel.

In a limited form, this allows you to monitor command-specific output in real time, but it comes with the following caveats:

• You'll have to manually activate the new windows individually to see the output being generated.

• The output is only visible while a command is running; on completion, its window closes automatically, so you can't inspect the output after the fact.

  • To work around that you'd have to use something like Tee-Object in your PowerShell cmdlet in order to also capture output in a file, which the caller could later inspect.

  • This is also the only way to make the output available programmatically, albeit only as text.

• Passing PowerShell commands to powershell.exe via Start-Process requires you to pass your commands as strings (rather than script blocks) and has annoying parsing requirements, such as the need to escape " chars. as \" (sic) - see below.

• Last and not least, using Start-Process also introduces significant processing overhead (though with very long-running commands that may not matter).

$startedAt = [datetime]::UtcNow

# Define the commands - of necessity - as *strings*.
# Note the unexpected need to escape the embedded " chars. as \"
$commands = '$n = 1; Start-Sleep $n; \"I ran for $n secs.\"',
            '$n = 2; Start-Sleep $n; \"I ran for $n secs.\"',
            '$n = 3; Start-Sleep $n; \"I ran for $n secs.\"'

# Use `Start-Process` to launch the commands asynchronously,
# in a new window each (Windows only).
# `-PassThru` passes an object representing the newly created process through.
$procs = $commands | ForEach-Object { Start-Process -PassThru powershell -Args '-c', $_ }

# Wait for all processes to exit.
$procs.WaitForExit()


"`nAll processes completed. Total runtime in secs.: $('{0:N2}' -f ([datetime]::UtcNow - $startedAt).TotalSeconds)"

---

^{[1] In Windows PowerShell v3 and v4, Install-Module isn't available by default, because these versions do not come with the PowerShellGet module. However, this module can be installed on demand, as detailed in Installing PowerShellGet}