Akka Stream Kafka vs Kafka Streams

Question

I am currently working with Akka Stream Kafka to interact with kafka and I was wonderings what were the differences with Kafka Streams.

I know that the Akka based approach implements the reactive specifications and handles back-pressure, functionality that kafka streams seems to be lacking.

What would be the advantage of using kafka streams over akka streams kafka?

Answer 1

Your question is very general, so I'll give a general answer from my point of view.

First, I've got two usage scenario:

1. cases where I'm reading data from kafka, processing it and writing some output back to kafka, for these I'm using kafka streams exclusively.
2. cases where either the data source or sink is not kafka, for those I'm using akka streams.

This already allows me to answer the part about back-pressure: for the 1st scenario above, there is a back-pressure mechanism in kafka streams.

Let's now only focus on the first scenario described above. Let's see what I would loose if I decided to stop using Kafka streams:

• some of my stream processors stages need a persistent (distributed) state store, kafka streams provides it for me. It is something that akka streams doesn't provide.
• scaling, kafka streams automatically balances the load as soon as a new instance of a stream processor is started, or as soon as one gets killed. This works inside the same JVM, as well as on other nodes: scaling up and out. This is not provided by akka streams.

Those are the biggest differences that matter to me, I'm hoping that it makes sense to you!

Answer 2

The big advantage of Akka Stream over Kafka Streams would be the possibility to implement very complex processing graphs that can be cyclic with fan in/out and feedback loop. Kafka streams only allows acyclic graph if I am not wrong. It would be very complicated to implement cyclic processing graph on top of Kafka streams