What does an HTTP 500 error with a 121 sub-status signify?

Question

I have a Web Application running in Azure and I've had a few outages where the server is unresponsive. When I look in the IIS log, I see HTTP 500 errors with a sc-substatus of 121 and a sc-win32-status of 0.

Omitting all other fields, the logs look like this, in this order:

sc-status sc-substatus sc-win32-status
500 121 0

I can't find reference to a 500.121 error anywhere online.

Answer 1

We saw the same issue.

TLDR

• We use Always Encrypted in an Azure Sql database.
• We use Azure Key Vault to store our master keys.
• We found the following bug logged for older versions of Azure Key Vault Provider - https://support.microsoft.com/en-us/help/4016853/fix-deadlock-when-apps-try-to-acquire-or-refresh-an-authentication-tok
• Solution - We upgraded the Azure Key Vault Provider from a 1.x version to the latest 2.1.0 version - https://www.nuget.org/packages/Microsoft.SqlServer.Management.AlwaysEncrypted.AzureKeyVaultProvider/2.1.0

Symptoms

• We began getting a 500/121 in production.
• All response times were just over 230 seconds.
• The issue always happened 2 hours apart... sometimes slightly longer.

Logging

• Our own logging showed nothing.
• ELMAH showed nothing.
• Failed Request Tracing showed nothing.
• The only indication was in the IIS logs.
• We also collected some .Net Profiler Traces from the Diagnose and solve problems tab in the App Service in the Azure portal and included the Thread Stacks. We noticed that there were a number of Thread Stacks that were doing stuff related to Always Encrypted. This was just a little curious, but it was the thing that finally got us looking at Always Encrypted as the culprit.

Reproduction

• This code was in 4 different environments.
• It was only happening in 2 of the environments.
• We had nothing that would cause it to happen.

MS Support Issue

• https://support.microsoft.com/en-us/help/4016853/fix-deadlock-when-apps-try-to-acquire-or-refresh-an-authentication-tok

Consider the following scenario:

You have Microsoft .NET Framework applications that use Always Encrypted in SQL Server 2016 or Azure SQL Database.
The column master keys for these applications are stored in the Azure Key Vault. 

In this scenario, the applications experience deadlocks. Therefore, the applications become unresponsive (hang) or time out.

The deadlocks may occur during attempts to acquire or refresh an authentication token for the Azure Key Vault.

Azure Key Vault Provider v2.1.0 Release Notes

• https://www.nuget.org/packages/Microsoft.SqlServer.Management.AlwaysEncrypted.AzureKeyVaultProvider/2.1.0

This release addresses a bug existing in all previous releases which can cause deadlocks in multi-threaded applications.

Answer 2

I just got this from one of the Azure Software Engineers:

121 is a timeout event which basically means that the request spent 230 seconds on the worker VM without initiating any read/write IO on the connection. It is highly likely for this to be an application issue, but doesn’t necessarily have to be so.

The IIS logs all have a time-taken value hovering around 230 seconds. Mystery solved.