For how long data is stored in kafka server?

Question

Suppose i am sending some messages from kafka producer to kafka consumer then where it will store? Is there any databases for storing messages ? And for how long messages are store?

Can anyone please explain it.

Answer 1

Data is produced/consumed to/from Kafka topics. A Topic is the core abstraction that Kafka provides for a stream of records. A topic is similar to a table in a typical database.

If you want to move the data from Kafka into a database (or vice versa) you can use Confluent's bundled connectors which can import and export data from some of the most commonly used data systems.

Regarding the period you can keep data in topics, you need to look for the retention policy and period.

The Kafka cluster durably persists all published records—whether or not they have been consumed—using a configurable retention period. For example, if the retention policy is set to two days, then for the two days after a record is published, it is available for consumption, after which it will be discarded to free up space.

Retention period is a configurable parameter that allows you to store the data for as long as you like. For example if you configure the following parameters;

log.retention.minutes=3
log.cleanup.policy=delete

a message will remain to the topic for 3 minutes. For more details regarding these parameters, have a look at the broker configuration parameters where you can also find information for some more parameters related to topic retention (such as log.retention.bytes, log.segment.bytes, log.retention.check.interval, log.roll.ms etc).

My advice for you would be to start with the introductory material which can be found in the official documentation in order to be able to understand how Kafka works in general