Put data first in Kafka or Database?

Question

What are te pro's and con's when putting data first in Kafka and then in a database, or the other way?

An example: A user does a REST (POST) call to store let's say products. Normally i would pick up this call in the backend and save the body to a database (after validation and all..). Is it a best practice to pick up this call and store de data in Kafka fist and then save it to the database (in this case, the database is a kafka consumer).

Or is it better to save it in the database first, then send it to kafka?

Thanks

Answer 1

lets take example of both scenarios with your use case, api call for storing a product lets say PRODUCT1 :

you database: product_table(product_id , product_name,product_info)

API pseudo code:

1. valiadteProductInfo
2. save - either first in kafka or in DB

APPROACH 1 -

saving to kafka first means that you might see that result in the DB some time later, you will return the product id to the user and if user wants to populate the product id its not visible. for me this is not the right approach as you will then need to handle many things on the UI side for such delay.

APPROACH 2 - saving to db first and kafka second there are two scenarios: 1. kafka push is sync in code- in this case in sending to kafka fails , which in your business case if very critical as other microservice is dependent. it is not the right approach , but if its ok that for <0.001 % of the time if push fails and then you delete the product from DB and return exception to the user. I think its completely ok on this.

2. kafka push is by polling the db for changes and put changes to kafka (read about EventSourcing for this) : in this case you will get 100% guarantee , but some small delay . this also you can use

Answer 2

I'd prefer to put Kafka as it has guarantee that message will not lost and it is durable. But if u will put 1st put to db then Kafka there is risk that your service can go down between writing to db and kafka.