How to Process a kafka KStream and write to database directly instead of sending it another topic

Question

I don't want to write processed KStream to another topic, I directly want to write enriched KStream to database. How should I proceed?

Answer 1

You can implement a custom Processor that opens a DB connection and apply it via KStream#process(). Cf. https://docs.confluent.io/current/streams/developer-guide/dsl-api.html#applying-processors-and-transformers-processor-api-integration

Note, you will need to do sync writes into your DB to guard against data loss.

Thus, not writing back to a topic has multiple disadvantages:

• reduced throughput because of sync writes
• you cannot use exactly-once semantics
• coupling your application with the database (if DB goes down, your app goes down, too, as it can't write its results anymore)

Therefore, it's recommended to write the results back into a topic and use Connect API to get the data into your database.