Why is Powershell's Install-Module so excruciatingly slow?

Question

I use Powershell 5.1 and we often install modules from our internal module repository hosted on our on-prem Azure Artifacts (we use Azure DevOps Server 2019).

The problem is that it is very slow. Takes regularly over 10 seconds to load one module. And this is not the network, which is pretty fast. It is the Install-Module internals.

I tried running Set-PSDebug -Trace 2 from within an Azure DevOps build in order to get line timestamps, but it is useless. For example, observe this output snippet:

2020-06-29T04:20:40.6944925Z DEBUG:  267+                 switch ( >>>> $MsgID)
2020-06-29T04:20:40.6957451Z DEBUG:     ! SET $switch = ''.
2020-06-29T04:20:40.6972578Z DEBUG:  290+  >>>> }                                      
2020-06-29T04:20:40.6986528Z DEBUG:     ! SET $switch = ''.
2020-06-29T04:20:40.6998323Z DEBUG:  232+                                              >>>> }  
2020-06-29T04:20:48.3791151Z DEBUG:  220+ $script:PackageManagementInstallModuleMessageResolverScriptBlock =   >>>> {
2020-06-29T04:20:48.3808676Z DEBUG:     ! CALL function '<ScriptBlock>'  (defined in file 'C:\Program 
2020-06-29T04:20:48.3811147Z Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1')
2020-06-29T04:20:48.3822332Z DEBUG:  222+                                                  >>>> $PackageTarget = 
2020-06-29T04:20:48.3824673Z $LocalizedData.InstallModulewhatIfMessage

It show 8 seconds pause, but the place where it shows it does not make any sense.

So, my question is this - why is it so slow? Is there a way to profile it reliably?

EDIT 1

Have just installed PS Core 7 - the same lousy performance for Install-Module. My version of PowershellGet is:

C:\> Get-Module PowershellGet | Select Version

Version
-------
2.2.4

C:\>

EDIT 2

Found this page - https://learn.microsoft.com/en-us/powershell/scripting/gallery/how-to/working-with-packages/manual-download?view=powershell-7 It explicitly warns against simulating Install-Module with nuget, even though it explains how to do it. I would like to understand more about the implications of using nuget instead of Install-Module, besides it working 5 times faster (on average).

EDIT 3

The modules are not signed. We are talking about our internal modules. But installing modules from PSGallery, like Az.Accounts, Az.Storage and Az.Sql takes about the same time. When our build needs to make sure 5 modules are installed it takes easily a minute. On another note, Install-Module is not concurrency safe, so when our builds were running it bare we were facing all kinds of weird errors. They went away when we introduced explicit named mutex around Install-Module. Needless to say, it did not contribute to performance.

Answer 1

While this doesn't answer your "why", you might like to look at JustinGrote's high performance Powershell Gallery Module Installer:

.SYNOPSIS
High Performance Powershell Module Installation

.DESCRIPTION
This is a proof of concept for using the Powershell Gallery OData 
API and HTTPClient to parallel install packages

It is also a demonstration of using async tasks in powershell 
appropriately. Who says powershell can't be fast?

This drastically reduces the bandwidth/load against Powershell 
Gallery by only requesting the required data

It also handles dependencies (via Nuget), checks for existing 
packages, and caches already downloaded packages

.NOTES
THIS IS NOT FOR PRODUCTION, it should be considered "Fragile" and 
has very little error handling and type safety

It also doesn't generate the PowershellGet XML files currently, so
PowershellGet will see them as "External" modules

It is indeed much faster.