Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Azure Function Event Hub Trigger reliability

I'm a bit confused regarding the EventHubTrigger for Azure functions.

I've got an IoT Hub, and am using its eventhub-compatible endpoint to trigger an Azure function that is going to process and store the received data.

However, if my function fails (= throws an exception), that message (or messages) being processed during that function call will get lost. I actually would expect the Azure function runtime to process the messages at a later time again. Specifically, I would expect this behavior because the EventHubTrigger is keeping checkpoints in the Function Apps storage account in order to keep track of where in the event stream it has to continue.

The documention of the EventHubTrigger even states that

If all function executions succeed without errors, checkpoints are added to the associated storage account

But still, even when I deliberately throw exceptions in my function, the checkpoints will get updated and the messages will not get received again.

Is my understanding of the EventHubTriggers documentation wrong, or is the EventHubTriggers implementation (or its documentation) wrong?

like image 438
Schweder Avatar asked Mar 08 '18 06:03

Schweder


People also ask

Which kind of Azure Functions should be used to create a reliable message queue system?

Azure Logic Apps or durable functions are a natural fit to manage the workflow and circuit state. Other services may work just as well, but logic apps are used for this example.

Can you manage event hub programmatically?

You can use the Azure Event Hubs management libraries to dynamically provision Event Hubs namespaces and entities. This dynamic nature enables complex deployments and messaging scenarios, so that you can programmatically determine what entities to provision.


1 Answers

This piece of documentation seems confusing indeed. I guess they mean the errors of Function App host itself, not of your code. An exception inside function execution doesn't stop the processing and checkpointing progress.

The fact is that Event Hubs are not designed for individual message retries. The processor works in batches, and it can either mark the whole batch as processed (i.e. create a checkpoint after it), or retry the whole batch (e.g. if the process crashed).

See this forum question and answer.

If you still need to re-process failed events from Event Hub (and errors don't happen too often), you could implement such mechanism yourself. E.g.

  1. Add an output Queue binding to your Azure Function.
  2. Add try-catch around processing code.
  3. If exception is thrown, add the problematic event to the Queue.
  4. Have another Function with Queue trigger to process those events.

Note that the downside of this is that you will loose ordering guarantee provided by Event Hubs (since Queue message will be processed later than its neighbors).

like image 77
Mikhail Shilkov Avatar answered Sep 22 '22 06:09

Mikhail Shilkov