Inexplicable storage transactions from Azure Functions

Question

I have a project with a couple of .NET Core based Azure Functions running on a schedule. One of them runs once every 10 minutes and serves to update view counts similar to how SO tracks question views and the other sends emails once a week. These functions were running just fine for a year or so. I recently updated them to use Azure Functions SDK v3 and Azure Functions runtime v3 as well as .NET Core 3.1 (basically moved from .NET Core 2.1 to .NET Core 3.1 so I needed to update the functions runtime).

At one point I received a bill much higher than usual. Turns out the functions which share the same underlying storage account began making A LOT of API transactions to the storage. Like many thousands each 5 minutes. Normally every run generates like 100 storage transactions (probably retrieving the function files?) but at some point the transactions jumped sharply. Upon restarting the functions the transaction drop to normal and everything is normal for a few days then they jump again and stay high until restart.

The functions code has not been changed with the upgrade only the SDK and the runtime. The function code has a constant number of log writes (like 7) through the SDK-provided logger and does not interact with the storage in any other way.

I have two identical environments one for test and one for production and both have the same problem. The interval that it takes for the function to go rogue is a couple of days but seems to be different each time. However if I restart both test and production at the same time the next spike happens simultaneously on both environments so there is something deterministic there.

According to my investigation through the Metrics tool the offending transaction types are Create, Close and ChangeNotify with some Cancel (but less than the others). The storage is not used for anything else (in fact it only exist because Azure Functions require a backing storage to store its files or something)

Here is the trigger code in case it is relevant

[FunctionName("ViewCountUpdater")]
public static async Task RunAsync([TimerTrigger("0 */10 * * * *"/*, RunOnStartup = true*/)]TimerInfo timer, ILogger log, ExecutionContext context)

I believe I have hit a bug with the Azure Functions runtime or the Azure Functions .NET Core SDK. Has anyone experienced this or know how to work around it?

Answer 1

After weeks of investigation the Azure support team and I think we have found the line that causes the issue and this is it:

.AddJsonFile("local.settings.json", optional: true, reloadOnChange: true)

The config file was not published as part of the publish process and is not present in Azure. Experiments for now seem to confirm that when this is present the transactions spike and when not they are normal. This does not answer

• Why the bug happens at all
• Is it a regression in .NET Core or in the functions runtime?
• Why does the bug happen randomly and not on every run?

Note that testing this takes time as I have to wait for days for the random spike and I can never be sure if it is gone forever so I am not 100% sure that at some point in the future the spike won't happen again and turn out that the issue was something else.

Answer 2

Do you have AzureWebJobsDashboard enabled? If yes, you should disable it (delete the Connection String from the App Settings) and switch to application insights. This setting has been known to cause unexpected writes to storage which can't be properly explained.

https://github.com/Azure/Azure-Functions/issues/832