Powershell Runspaces Parallel script execution: Is Set-AzureRmSqlDatabaseTransparentDataEncryption commandlet threadsafe?

Question

When running the powershell commandlet: Set-AzureRmSqlDatabaseTransparentDataEncryption

in parallel and separate runspaces, we are seeing transient/intermittent failures with stacktrace seen below. However, when we run this same commandlet serially, I've not seen this issue happen.

Possible Reasons I've come up with so far:

• Is it known that this commandlet is not thread-safe? (on purpose or bug?)
• Contention/locking on file {ReleaseUser}\AppData\Roaming\Windows Azure Powershell\ErrorRecords\Set-AzureRmSqlDatabaseTransparentDataEncryption_YYYY-MM-DD-THH-MM-SS-PPP.log where the commandlet seems to log information related to resolving a token (no actual errors in this file).

Stack Trace for one of the scripts running this command:

[Exception:System.NullReferenceException: Object reference not set to an instance of an object.\r\n
at System.Collections.Specialized.OrderedDictionary.OrderedDictionaryEnumerator.get_Value()\r\n
at Microsoft.Azure.Commands.Common.Authentication.Factories.ClientFactory.GetCustomHandlers()\r\n
at Microsoft.Azure.Commands.Common.Authentication.Factories.ClientFactory.CreateClient[TClient](AzureContext context, Endpoint endpoint)\r\n
at Microsoft.Azure.Commands.Sql.TransparentDataEncryption.Services.AzureSqlDatabaseTransparentDataEncryptionCommunicator.GetCurrentSqlClient(String clientRequestId)\r\n
at Microsoft.Azure.Commands.Sql.TransparentDataEncryption.Adapter.AzureSqlDatabaseTransparentDataEncryptionAdapter.GetTransparentDataEncryption(String resourceGroupName, String serverName, String databaseName)\r\n
at Microsoft.Azure.Commands.Sql.TransparentDataEncryption.Cmdlet.SetAzureSqlDatabaseTransparentDataEncryption.GetEntity()\r\n
at Microsoft.Azure.Commands.Sql.Common.AzureSqlCmdletBase'2.ExecuteCmdlet()\r\n
at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()]\

Stack Trace For a second thread running the same commandlet:

Message: "Object reference not set to an instance of an object.", Source: "System", StackTrace: " at System.Collections.Specialized.OrderedDictionary.IndexOfKey(Object key)\r\n at System.Collections.Specialized.OrderedDictionary.set_Item(Object key, Object value)\r\n at Microsoft.Azure.Commands.Common.Authentication.Factories.ClientFactory.AddHandler[T](T handler)\r\n at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.BeginProcessing()\r\n at System.Management.Automation.Cmdlet.DoBeginProcessing()\r\n at System.Management.Automation.CommandProcessorBase.DoBegin()"`

Suspicious: both commands blow up when enumerating/accessing System.Collections.Specialized.OrderedDictionary according to the stack-traces.

Significance: Are the two instances of the command accessing the SAME dictionary?

Answer 1

Will's answer seems to support our suspicion that this (and other) commandlets are not thread-safe. In order to deal with this, we "lock" around calling such commandlets using a mutex as the coordination mechanism.

Using this thread as inspiration, built a simple implementation of a process locking/coordination mechanism (using Mutexes) in powershell:

function Wait-OnMutex
{
    param(
    [parameter(Mandatory = $true)][string] $MutexId
    )

    try
    {
        $MutexInstance = New-Object System.Threading.Mutex -ArgumentList 'false', $MutexId

        while (-not $MutexInstance.WaitOne(1000))
        {
            Start-Sleep -m 500;
        }

        return $MutexInstance
    } 
    catch [System.Threading.AbandonedMutexException] 
    {
        $MutexInstance = New-Object System.Threading.Mutex -ArgumentList 'false', $MutexId
        return Wait-OnMutex -MutexId $MutexId
    }

}

##
## example script calling unsafe commandlet
##

$MutexInstance = Wait-OnMutex -MutexId 'SomeMutexId12345'

## this is where you do work inside the "lock"
## call the commandlet needing to be single-threaded
Set-AzureRmSqlDatabaseTransparentDataEncryption -arg1 value

$MutexInstance.ReleaseMutex()

Hope This helps somebody.