Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How can I check how much disk space is being used by Kafka

Tags:

apache-kafka

My disk space is running out, after syncing 2 millions message in one of my Kafka topic. I even deleted the topic but still the space is somehow seems consumed. Any help of link would be appreciated.

like image 849
Nadeem Avatar asked Jun 03 '16 22:06

Nadeem


People also ask

How much space does Kafka use?

Furthermore, Kafka uses heap space very carefully and does not require setting heap sizes more than 6 GB. This will result in a file system cache of up to 28-30 GB on a 32 GB machine. You need sufficient memory to buffer active readers and writers.

Does Kafka use disk?

The Disk Usage metric shows the percentage of disk space being used by Kafka. Kafka will gradually use more and more disk space in order to store messages for each topic.

What happens if Kafka partition is full?

Often the data will stay there and get deleted once a specified retention period or max size/data limit has been reached.


2 Answers

Use the below command to know how much disk space consumed by each topic

du -sh /tmp/kafka-logs/*
like image 51
Kamal Chandraprakash Avatar answered Nov 03 '22 00:11

Kamal Chandraprakash


Another way to determine the size of the topic to send metrics, such as graphite. Then the size can be found as follows: kafka.log.Log.<topic_name>-<partition_number>-Size.value

like image 25
Andrey Avatar answered Nov 02 '22 23:11

Andrey