Azure Function App delay retry for azure service bus

Question

First let me explain what I have. I have myself an Azure Service Bus with an Azure Function App. The Service Bus is setup to use SQL Filters to push specific message types into specific topics. Then using my Azure Function App these will get the newest message and then process it.

A basic example

1: I send a request to my EmailAPI

2: EmailAPI then pushing a new message into the Service Bus with a type of "Email"

3: The SQL Filter then sees the type is of "Email" and is placed into the email Topic in the Service Bux

4: The EmailListener Azure Function monitors the Service bus and notices a new message

5: Gather the Service Bus message and process it (basically just send the email using the information provided)

Now let's say for some reason the SMTP server connection is a little broken and some times we get a TimeOutException when attempting to send the email (EmailListener). What happens now when an exception is thrown, the Function App EmailListener will attempt to send it again instantly, no wait, it will just attempt to send it again. It will do this for a total of 10 times and then inform the Service Bus to place the message in the Dead Letter queue.

What I am attempting to do is when an exception is thrown (such as TimeOutException), we wait X amount of time before attempting to process the same message again. I have looked around at many different posts talking about the host.json and attempting to set those settings, but these have not worked. I have found a solution, however the solution requires your to create a clone of the message and push it back into the Service Bus and give it a delayed process time. I would prefer not to implement my own manual delay system, if Azure Service Bus / Function App can deal with retries itself.

The biggest issue I am having (which is probably down to my understanding) is who is at fault? Is it the Service Bus settings to handle the Retry Policy or is it the Azure Function App to deal with attempting to retry after X time.

I have provided a some code, but I feel code isn't really going to help explain my question.

// Pseudo code
public static class EmailListenerTrigger
{
    [FunctionName("EmailListenerTrigger")]
    public static void Run([ServiceBusTrigger("messages", "email", Connection = "ConnectionString")]string mySbMsg, TraceWriter log)
    {
           var emailLauncher = new EmailLauncher("SmtpAddress", "SmtpPort", "FromAddress");
           try
           {
               emailLauncher.SendServiceBusMessage(mySbMsg);
           }
           catch(Exception ex)
           {
               log.Info($"Audit Log: {mySbMsg}, Excpetion: {ex.message}");
           }
    }
}

reference one: https://blog.kloud.com.au/2017/05/22/message-retry-patterns-in-azure-functions/ (Thread.Sleep doesn't seem like a good idea)

reference two: https://github.com/Azure/azure-functions-host/issues/2192 (Manually implemented retry)

reference three: https://www.feval.ca/posts/function-queue-retry/ (This seems to refer to queues when I am using topics)

reference four: Can the Azure Service Bus be delayed before retrying a message? (Talks about Defering the message, but then you need to manually get it back out the queue/topic.)

Answer 1

You might be able to solve your issue with the use of Durable Functions. There is for example a built-in method CallActivityWithRetryAsync() that can retry when the activity functions throws an exception.

https://learn.microsoft.com/en-us/sandbox/functions-recipes/durable-diagnostics#calling-activity-functions-with-retry

Your flow would probably something like this:

1. Service Bus triggered Function. This one starts an Orchestrator Function

2. The orchestrator calls your activity function (using the aforementioned method)

3. Your email sending is implemented in an Activity Function and can throw exceptions as needed

Answer 2

While there is no native support for what you want to do, it is still doable without having to do a lot of custom development. You can basically add a service bus output binding to your Azure function, that is connected to the same queue your function consumes messages from. Then, use a custom property to track the number of retries. The following is an example:

private static TimeSpan[] BackoffDurationsBetweenFailures = new[] { }; // add delays here

[FunctionName("retrying-poc")]
public async Task Run(
  [ServiceBusTrigger("myQueue")] Message rawRequest,
  IDictionary<string, object> userProperties,
  [ServiceBus("myQueue")] IAsyncCollector<Message> collector)
{
  var request = GetRequest(rawRequest);
  var retryCount = GetRetryCount(userProperties);
  var shouldRetry = false;

  try
  {
    await _unreliableService.Call(request);
  }
  catch (Exception ex)
  {
     // I don't retry if it is a timeout, but that's my own choice.
    shouldRetry = !(ex is TimeoutException) && retryCount < BackoffDurationsBetweenFailures.Length;
  }

  if (shouldRetry)
  {
    var retryMessage = new Message(rawRequest.Body);
    retryMessage.UserProperties.Add("RetryCount", retryCount + 1);
    retryMessage.ScheduledEnqueueTimeUtc = DateTime.UtcNow.Add(BackoffDurationsBetweenFailures[retryCount]);
    await collector.AddAsync(retryMessage);
  }
}


private MyBusinessObject GetRequest(Message rawRequest)
  => JsonConvert.DeserializeObject<MyBusinessObject>(Encoding.UTF8.GetString(rawRequest.Body));

private int GetRetryCount(IDictionary<string, object> properties)
  => properties.TryGetValue("RetryCount", out var value) && int.TryParse(value.ToString(), out var retryCount)
            ? retryCount
            : 0;