How to execute a lot of durable functions triggered by Azure Queue?

Question

If briefly, our task is to process a lot of input messages.

To solve this we decided to use Azure Queue Storage and Azure Functions. We have Azure Functions structure similar to the following code:

Queue triggered function

[FunctionName("MessageControllerExecutor")]
public static async void Run(
    [QueueTrigger(QUEUE_NAME, Connection = QUEUE_CONNECTION_NAME)]string queueMessage,
    [OrchestrationClient] DurableOrchestrationClient client,
    TraceWriter log)
{
    await client.StartNewAsync("MessageController", queueMessage);
}

Durable function

[FunctionName("MessageController")]
public static async void Run(
    [OrchestrationTrigger] DurableOrchestrationContext context,
    TraceWriter log)
{
    if (!context.IsReplaying) log.Warning("MessageController started");

    var function1ResultTask = context.CallActivityAsync<ResultMessage>("Function_1", new InputMessage());
    var function2ResultTask = context.CallActivityAsync<ResultMessage>("Function_2", new InputMessage());

    await Task.WhenAll(function1ResultTask, function2ResultTask);

    // process Function_1 and Function_2 results
    // ...
}

Simple activity function sample

[FunctionName("Function_1")]
public static ResultMessage Run(
    [ActivityTrigger] DurableActivityContext activityContext,
    TraceWriter log)
{
    var msg = activityContext.GetInput<InputMessage>();
    int time = new Random().Next(1, 3);
    Thread.Sleep(time * 1000);

    return new ResultMessage()
    {
        Payload = $"Function_1 slept for {time} sec"
    };
}

MessageControllerExecutor triggered when a new item is received in a queue. MessageController is a Durable Function that uses a few simple activity functions to process each message.

When we push messages to the queue, the MessageControllerExecutor function starts immediately and asynchronously fires the MessageController and passes the message, so this works as expected.

But we are faced with the problem. Not all MessageController function instances run.

For example, we pushed 100 messages into the queue, but only about 10-20% of the messages were processed by MessageController.

Some messages were not processed or were processed with a long delay. It looks like durable functions failed to start б, though no exceptions were thrown.

We have a few questions:

1. Is this solution with queue triggered and durable functions correct to process the message queue or there is a better way to trigger durable functions by the queue?
2. Are there any limitations to run durable functions?
3. How many durable functions can be executed at the same time?

Answer 1

1. Yes this is a totally valid way to kick off orchestrations!
2. Sure, here's some details on the architecture as it relates to performance and scalability.
3. I think what you're probably intending to ask here is: how many orchestration instances of a single durable function definition can be executed at the same time? This is indeed a very important aspect to understand. Orchestration functions themselves are single threads and, per that link on scale I gave you above, are balanced across a set of control queues. You can read the document for more information, but the bottom line is you don't want to do any work other than actual orchestration in your orchestration function because they are your limit on scalability. It is the orchestration action functions which behave like any other Azure Function and have virtually no limits on their scalability.

You did elide some code from your orchestration trigger in the question above for the purposes of brevity which I understand, but what exactly are you doing there after the await Task.WhenAll(...)? If it includes any kind of significant processing you should really be farming that out to a third action function (e.g. Function_3) to do and then simply returning the results from the orchestration function.

Update: I just noticed your functions are defined as async void. If I had to guess, this would actually cause a problem for the runtime. Can you try changing it to async Task and see if your problem goes away? As a general rule defining methods as async void is frowned upon in .NET.

Answer 2

Some extension for Drew's answer. You should not use Thread.Sleep(), as the documentation states, instead use CreateTimer Api.