Azure Durable Framework Function App VERY Slow

Question

I have made an app that uses azure functions durable fan-out strategy to make parallel inquiries and updates to a database by sending http requests to our own internal API.

I found out that the fan-out strategy is EXTREMELY slower than it would be to use TPL library and doing parallelism this way on a normal .net Core webapp. It's not just slower it's about 20 times slower. It takes 10 minutes for 130 updates while the .net core 3.1 app that I've made for speed comparisson that does the exact same thing does 130 updates in 0.5 minutes and in a significantly lower plan.

I understand there is a latency becuase of the durable framwork infrastructure (communicating with the storage account and whatnot) but I don't see how that speed difference is normal. Each individual update happens in an ActivityTrigger function and the orchestrator is the one that gathers all the necessary updates and puts them in a Task.WhenAll() call just like the example from Microsoft docs.

Am I doing something wrong here? Is this business scenario maybe not compatible with this technology? The code seems to work fine and the parallelism works It's just a LOT slower than the .net core app. Another thing to mention is that the moment the function opens a second instance (due to either it being in consuption plan and naturally openning a second instance to deal with heavy load or it being in appservice plan and I manually opening an instance) it goes even slower although the cpu load somehow balances in the two instances. I suspect this could be extra latency due to azure queue comunication between the two instances but I'm not entirely sure.

One last detail is that the app also has a TimeTrigger that does a simple select in a database every one minute (nothing even remotely cpu intensive but it might play a role in the performance).

I've tried the function app in a premium plan, consuption plan, and appservice plan and it seems to top at 130 updates in 10 minutes no matter how huge the plan is.

Answer 1

Speaking generally, TPL will almost always be much faster than Durable Functions because all the coordination is done in-memory (assuming to don't completely exhaust system resources doing everything on one machine). So that part is often expected. Here are a few points worth knowing:

• Each fan-out to an activity function involves a set of queue transactions: one message for calling the activity function and one message for handing the result back to the orchestrator. When there are multiple VMs involved, then you also have to worry about queue polling delays.
• By default, the per-instance concurrency for activity functions is limited to 10 on a single-core VM. If your activity functions don't require much memory or CPU, then you'll want to crank up this value to increase per-instance concurrency.
• If you're using the Azure Functions Consumption or Premium plans, it will take 15-30 seconds before new instances get added for your app. This matters mainly if your workload can be done faster by running on multiple machines. The amount of time a message spends waiting on a queue is what drives scale-out (1 second is considered too long).

You can find more details on this in the Durable Functions Performance and Scale documentation.

One last thing I will say is the key value add of Durable Functions is orchestrating work in a reliable way in a distributed environment. However, if your workload isn't long-running, doesn't require strict durability/resilience, doesn't require scale-out to multiple VMs, and if you have strict latency requirements, then Durable Functions might not be the right tool. If you just need a single VM and want low latency, then a simple function that uses in-memory TPL may be a better choice.