Using Kafka to send batch emails

Question

So I've loved the idea of Kafka since I first heard of it but I haven't had the opportunity to get hands-on with it until recently. I think I have a use case that might apply but I'd like to get some opinions from people who are more familiar with it.

Basically I'm thinking about a notification system that would batch messages over a given period of time (say 30 minutes) and send them out as emails, in-app notifications, or otherwise. I like Kafka for this problem primarily because of its inherent durability. I had considered using a more straightforward message queue like RabbitMQ, ActiveMQ, SQS, etc. but I don't like that it would force me to manage buffering on the consumer-side and risk losing messages. Otherwise I would have to buffer in a secondary durable store which seems to defeat the purpose of having the queue in the first place.

So my idea would be to group the notifications in partitions by user and then every 30 minutes the consumer would read the last 30 minutes of data, aggregate it, and send a summary notification composed of the individual notifications.

I have a few concerns:

1. Am I crazy for thinking this is a good use case? With a little googling I don't see lots of people talking about using Kafka for exactly this purpose but it seems so perfect to me.
2. How should I handle individual notification errors? Say for example a user gets 50 notifications in a 30 minute window that will get grouped into 3 distinct messages to be sent out separately. Let's say two succeed but one fails, how should I handle retry logic? I've found some relatively new/obscure stuff like this https://github.com/softwaremill/kmq that seems to try to address the issue but I'm a little worried that I'm worried this just doesn't fit the Kafka model.
3. Am I just going against the grain? Surely this is a solved problem that people build everyday. Is there a more easy and obvious technology that I'm overlooking?

Thanks for your feedback!

Answer 1

It might be too late to answer this question now and I think you might have a solution already. For other users who are thinking about the same thing, I'd like to say that your idea is pretty good especially when considering using Kafka Streams. I am building a project called light-email now with Kafka Streams and Kotlin. Currently, I am thinking to send out email per event; however, it would be very easy to aggregate multiple events together within a time window in Kafka Streams.

To clarify two points from the comments.

1. We don't need to create a partition per user. Just need to ensure that the events belong to the same user goes to the same partition. This simply means that we need to hash the userid to load balance between partitions.

2. When message sending fails, it should be moved to the dead letter topic to process later. This is to prevent the current topic is blocked.