Running multiple scriptblocks at the same time with Start-Job (instead of looping)

Question

Hi all!

I've been looking for a way to make my script more efficient and I've come to the conclusion (with help from the nice people here on StackOverflow) that Start-Job is the way to go.

I have the following foreach-loop that I would like to run simultanously on all the servers in $servers. I have problems understanding how I actually collect the information returned from Receive-Job and add to $serverlist.

PS: I know that I am far away from getting this nailed down, but I would really appreciate some help starting out as I am quite stumped on how Start-Job and Receive-Job works..

# List 4 servers (for testing)
$servers = Get-QADComputer -sizelimit 4 -WarningAction SilentlyContinue -OSName *server*,*hyper*

# Create list
$serverlistlist = @()

# Loop servers
foreach($server in $servers) {

    # Fetch IP
    $ipaddress = [System.Net.Dns]::GetHostAddresses($Server.name)| select-object IPAddressToString -expandproperty IPAddressToString

    # Gather OSName through WMI
    $OSName = (Get-WmiObject Win32_OperatingSystem -ComputerName $server.name ).caption

    # Ping the server
    if (Test-Connection -ComputerName $server.name -count 1 -Quiet ) {
        $reachable = "Yes"
    }

    # Save info about server
    $serverInfo = New-Object -TypeName PSObject -Property @{
        SystemName = ($server.name).ToLower()
        IPAddress = $IPAddress
        OSName = $OSName
    }
    $serverlist += $serverinfo | Select-Object SystemName,IPAddress,OSName
}

Notes

• I am outputting $serverlist to a csv-file at the end of the script
• I list aprox 500 servers in my full script

Answer 1

Since your loop only needs to work with a string it's easy to turn it into a concurrent script.

Below is an example of making making your loop use background jobs to speed up processing.

The code will loop through the array and spin up background jobs to run the code in the script block $sb. The $maxJobs variable controls how many jobs run at once and the $chunkSize variable controls how many servers each background job will process.

Add the rest of your processing in the script block adding whatever other properties you want to return to the PsObject.

$sb = {
    $serverInfos = @()
    $args | % {
        $IPAddress = [Net.Dns]::GetHostAddresses($_) | select -expand IPAddressToString
        # More processing here... 
        $serverInfos += New-Object -TypeName PsObject -Property @{ IPAddress = $IPAddress }
    }
    return $serverInfos
}

[string[]] $servers = Get-QADComputer -sizelimit 500 -WarningAction SilentlyContinue -OSName *server*,*hyper* | Select -Expand Name

$maxJobs = 10 # Max concurrent running jobs.
$chunkSize = 5 # Number of servers to process in a job.
$jobs = @()

# Process server list.
for ($i = 0 ; $i -le $servers.Count ; $i+=($chunkSize)) {
    if ($servers.Count - $i -le $chunkSize) 
        { $c = $servers.Count - $i } else { $c = $chunkSize }
    $c-- # Array is 0 indexed.

    # Spin up job.
    $jobs += Start-Job -ScriptBlock $sb -ArgumentList ( $servers[($i)..($i+$c)] ) 
    $running = @($jobs | ? {$_.State -eq 'Running'})

    # Throttle jobs.
    while ($running.Count -ge $maxJobs) {
        $finished = Wait-Job -Job $jobs -Any
        $running = @($jobs | ? {$_.State -eq 'Running'})
    }
}

# Wait for remaining.
Wait-Job -Job $jobs > $null

$jobs | Receive-Job | Select IPAddress

Here is the version that processes a single server per job:

$servers = Get-QADComputer -WarningAction SilentlyContinue -OSName *server*,*hyper*

# Create list
$serverlist = @()

$sb = {
    param ([string] $ServerName)
    try {
        # Fetch IP
        $ipaddress = [System.Net.Dns]::GetHostAddresses($ServerName)| select-object IPAddressToString -expandproperty IPAddressToString

        # Gather OSName through WMI
        $OSName = (Get-WmiObject Win32_OperatingSystem -ComputerName $ServerName ).caption

        # Ping the server
        if (Test-Connection -ComputerName $ServerName -count 1 -Quiet ) {
            $reachable = "Yes"
        }

        # Save info about server
        $serverInfo = New-Object -TypeName PSObject -Property @{
            SystemName = ($ServerName).ToLower()
            IPAddress = $IPAddress
            OSName = $OSName
        }
        return $serverInfo
    } catch {
        throw 'Failed to process server named {0}. The error was "{1}".' -f $ServerName, $_
    }
}

# Loop servers
$max = 5
$jobs = @()
foreach($server in $servers) {
    $jobs += Start-Job -ScriptBlock $sb -ArgumentList $server.Name
    $running = @($jobs | ? {$_.State -eq 'Running'})

    # Throttle jobs.
    while ($running.Count -ge $max) {
        $finished = Wait-Job -Job $jobs -Any
        $running = @($jobs | ? {$_.State -eq 'Running'})
    }
}

# Wait for remaining.
Wait-Job -Job $jobs > $null

# Check for failed jobs.
$failed = @($jobs | ? {$_.State -eq 'Failed'})
if ($failed.Count -gt 0) {
    $failed | % {
        $_.ChildJobs[0].JobStateInfo.Reason.Message
    }
}

# Collect job data.
$jobs | % {
    $serverlist += $_ | Receive-Job | Select-Object SystemName,IPAddress,OSName
}

Answer 2

Something you need to understand about Start-Job is that it starts a new instance of Powershell, running as a separate process. Receive-job gives you a mechanism to pull the output of that session back into your local session to work with it in your main script. Attractive as it might sound, running all of those simultaneously would mean starting up 500 instances of Powershell on your computer, all running at once. That's probably going to have some unintended consequences.

Here's one way to approach dividing up the work, if it helps:

Splits up a array of computer names into $n arrays, and starts a new job using each array as the argument list to the script block:

  $computers = gc c:\somedir\complist.txt
  $n = 6
  $complists = @{}
  $count = 0
  $computers |% {$complists[$count % $n] += @($_);$count++}

  0..($n-1) |% {
  start-job -scriptblock {gwmi win32_operatingsystem -computername $args} - argumentlist $complists[$_]
  }