Message routing in kafka

Question

We're trying to build a platform using microservices that communicate async over kafka. It would seem natural, the way i understood it, to have 1 topic per aggregate type in each microservice. So a microservice implementing user registration would publish user related events into the topic "users". Other microservices would listen to events created from the "users" microservices and implement their own logic and fill their DBs accordingly. The problem is that other microservices might not be interested in all the events generated by the user microservice but rather a subset of these events, like UserCreated only (without UsernameChanged... for example). Using RabbitMq is easy since event handlers are invoked based on message type.

1. Did you ever implement message based routing/filtering over kafka?
2. Should we consume all the messages, deserialize them and ignore unneeded ones by the consumer? (sounds like an overhead)
3. Should we forward these topics to storm and redirect these messages to consumer targeted topics? (sounds like an overkill and un-scalable)
4. Using partitions doesn't seem logical as a routing mechanism

Answer 1

Use a different topic for each of the standard object actions: Create, Read, Update, and Delete, with a naming convention like "UserCreated", "UserRead", etc. If you think about it, you will likely have a different schema for the objects in each. Created will require a valid object; Read will require some kind of filter; Update you might want to handle incremental updates (add 10 to a specific field, etc).

If the different actions have different schemas it makes deserialization difficult. If you're in a loosey-goosey language like JavaScript, ok -- no big deal. But a strictly typed language like Scala and having different schemas in this same topic is problematic.

It'll also solve you're problem -- you can listen for exactly the types of actions you want, no more, not less.