Is it safe to use async helper functions in an Azure Durable Functions Orchestator?

Question

I am trying to track down some occasional Non-Deterministic workflow detected: TaskScheduledEvent: 0 TaskScheduled ... errors in a durable function project of ours. It is infrequent (3 times in 10,000 or so instances).

When comparing the orchestrator code to the constraints documented here there is one pattern we use that I am not clear on. In an effort to make the orchestrator code more clean/readable we use some private async helper functions to make the actual CallActivityWithRetryAsync call, sometimes wrapped in an exception handler for logging, then the main orchestrator function awaits on this helper function.

Something like this simplified sample:

[FunctionName(Name)]
public static async Task RunPipelineAsync(
    [OrchestrationTrigger]
    DurableOrchestrationContextBase context,

    ILogger log)
{
    // other steps

    await WriteStatusAsync(context, "Started", log);

    // other steps

    await WriteStatusAsync(context, "Completed", log);
}

private static async Task WriteStatusAsync(
    DurableOrchestrationContextBase context,
    string status,
    ILogger log
)
{
    log.LogInformationOnce(context, "log message...");
    try
    {
        var request = new WriteAppDocumentStatusRequest 
        {
            //...
        };

        await context.CallActivityWithRetryAsync(
            "WriteAppStatus",
            RetryPolicy,
            request
        );
    }
    catch(Exception e)
    {
        // "optional step" will log errors but not take down the orchestrator
        // log here
    }
}

In reality these tasks are combined and used with Task.WhenAll. Is it valid to be calling these async functions despite the fact that they are not directly on the context?

Answer 1

Yes, what you're doing is perfectly safe because it still results in deterministic behavior. As long as you aren't doing any custom thread scheduling or calling non-durable APIs that have their own separate async callbacks (for example, network APIs typically have callbacks running on a separate thread), you are fine.

If you are ever unsure, I highly recommend you use our Durable Functions C# analyzer to analyzer your code for coding errors. This will help flag any coding mistakes that could result in Non-deterministic workflow errors.

UPDATE

Note: the current version of the analyzer will require you to add a [Deterministic] attribute to your private async function, like this:

[Deterministic]
private static async Task WriteStatusAsync(
    DurableOrchestrationContextBase context,
    string status,
    ILogger log
)
{
   // ...
}

This lets it know that the private async method is being used by your orchestrator function and that it also needs to be analyzed. If you're using Durable Functions 1.8.3 or below, the [Deterministic] attribute will not exist. However, you can create your own custom attribute with the same name and the analyzer will honor it. For example:

[Deterministic]
private static async Task WriteStatusAsync(
    DurableOrchestrationContextBase context,
    string status,
    ILogger log
)
{
   // ...
}

// Needed for the Durable Functions analyzer
class Deterministic : Attribute { }

Note, however, that we are planning on removing the need for the [Deterministic] attribute in a future release, as we're finding it may not actually be necessary.