How to get rid of old messages from Azure EventHub?

Question

When we subscribe to an EventHub for a particular type. I see my worker keeps processing the old messages as well. Is there any way to get rid of older messages in the event hub or change the partition's offset value at a desired message manually? As of now our worker doesn't keep track of processed messages when the worker reboots the Checkpoint is not save and the old messages are re-processed... So Can anyone share a way to get rid of this issue?

Thanks in Advance, Radha.

Answer 1

You cannot delete messages from an Azure Event Hub, but I'm going to provide a couple of alternatives.

1. You can shorten your message retention period. By default you can set your retention period to keep messages between 1-7 days. If it's holding more data than it is relevant, simply shorten it.
2. Each message that enters an Azure Event Hub is appended with a column/attribute called "EventEnqueuedUtcTime", which represents the time it entered the Event Hub. Have your program subscribe to the Event Hub. All the events will stream in, but you can program in an IF statement to only process the messages that have an EventEnqueuedUtcTime after a specified time.Keep in mind this time is the server time of your hosted Namespace, who's clock may be different from the time of measurement.
3. Append your own time-stamp to your messages and do step 2 with this new attribute/column.

Answer 2

Imagine EventHub as a persistent Stream of events stored on cloud. You will need to maintain the Cursor of where to start reading from. There is no other way around. You will need to checkpoint to be able to do this. Most of the Messaging users confuse with the Topic semantics in EventHubs. This existing question can bring in clarity in that direction.

more on Event Hubs...