How to implement exponential backoff in Azure Functions?

Question

How to implement exponential backoff in Azure Functions?

I have a function that depends on external API. I would like to handle the unavailability of this service using the retry policy. This function is triggered when a new message appears in the queue and in this case, this policy is turned on by default:

For most triggers, there is no built-in retry when errors occur during function execution. The two triggers that have retry support are Azure Queue storage and Azure Blob storage. By default, these triggers are retried up to five times. After the fifth retry, both triggers write a message to a special poison queue.

Unfortunately, the retry starts immediately after the exception (TimeSpan.Zero), and this is pointless in this case, because the service is most likely still unavailable. Is there a way to dynamically modify the time the message is again available in the queue?

I know that I can set visibilityTimeout (host.json reference), but it's set for all queues and that is not what I want to achieve here.

I found one workaround, but it is far from ideal solution. In case of exception, we can add the message again to the queue and set visibilityTimeout for this message:

[FunctionName("Test")]
public static async Task Run([QueueTrigger("queue-test")]string myQueueItem, TraceWriter log,
    ExecutionContext context, [Queue("queue-test")] CloudQueue outputQueue)
{
    if (true)
    {
        log.Error("Error message");
        await outputQueue.AddMessageAsync(new CloudQueueMessage(myQueueItem), TimeSpan.FromDays(7),
            TimeSpan.FromMinutes(1), // <-- visibilityTimeout
            null, null).ConfigureAwait(false);
        return;
    }
}

Unfortunately, this solution is weak because it does not have a context (I do not know which attempt it is and for this reason I can not limit the number of calls and modify the time (exponential backoff)).

Internal retry policy also is not welcome, because it can drastically increase costs (pricing models).

Answer 1

Microsoft added retry policies around November 2020 (preview), which support exponential backoff:

[FunctionName("Test")]
[ExponentialBackoffRetry(5, "00:00:04", "00:15:00")] // retries with delays increasing from 4 seconds to 15 minutes
public static async Task Run([QueueTrigger("queue-test")]string myQueueItem, TraceWriter log, ExecutionContext context)
{
    // ...
}

Answer 2

I had a similar problem and ended up using durable functions which have an automatic retry feature built-in. This can be used when you wrap your external API call into activity and when calling this activity you can configure retry behavior through the options object. You can set the following options:

Max number of attempts: The maximum number of retry attempts.

First retry interval: The amount of time to wait before the first retry attempt.

Backoff coefficient: The coefficient used to determine rate of increase of backoff. Defaults to 1.

Max retry interval: The maximum amount of time to wait in between retry attempts.

Retry timeout: The maximum amount of time to spend doing retries. The default behavior is to retry indefinitely.

Handle: A user-defined callback can be specified to determine whether a function should be retried.