See size of Kafka Topics in Bytes

Question

For Metrics we meed to see the total size of a Kafka Topic in bytes across all partitions and brokers.

I have been searching for quite a while on how to do this and I haven't worked out if this is possible and how to do it.

We are on V0.82 of Kafka.

Answer 1

You can see the partition size using the script /bin/kafka-log-dirs.sh

/bin/kafka-log-dirs.sh --describe --bootstrap-server <KafakBrokerHost>:<KafakBrokerPort> --topic-list <YourTopic>

Answer 2

As Martbob very helpfully mentioned, you can do this using kafka-log-dirs. This produces JSON output (on one of the lines). So I can use the ever-so-useful jq tool to pull out the 'size' fields (some are null), select only the ones that are numbers, group them into an array, and then add them together.

kafka-log-dirs \
    --bootstrap-server 127.0.0.1:9092 \
    --topic-list 'topic_of_interest' \
    --describe \
  | grep '^{' \
  | jq '[ ..|.size? | numbers ] | add'

Example output: 67704

I haven't verified if the output makes sense, so you should check that yourself.