kafka loses all topics on reboot

Question

I'm trying out Kafka (0.8.2.1) in a VM, but am having trouble with it: though everything is fine while the machine remains on (even if I restart ZK/Kafka), if I reboot the machine (after gracefully shutting down ZK/Kafka) it seems all Kafka topics go lost.

I'm probably missing something basic here, since this is probably not supposed to happen. What might it be?

cd /vagrant/kafka_2.11-0.8.2.1
bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 10 --topic foo
bin/kafka-topics.sh --list --zookeeper localhost:2181
# foo
# ^C then resume ZooKeeper, Kafka, or both
bin/kafka-topics.sh --list --zookeeper localhost:2181
# foo
# ^C both, reboot machine, boot ZK/Kafka again
bin/kafka-topics.sh --list --zookeeper localhost:2181
# no topics

Answer 1

Looks like the default location for logs is in the /tmp directory which gets wiped on reboot. Change that location in the config to a more permanent location.

Answer 2

This happens because the tmp folder get cleared out on reboot.

To fix this issue, do the following.

Go to you kafka installation directory and search for the file server.properties. You should see a section as below

A comma separated list of directories under which to store log files
log.dirs=/tmp/kafka-logs

Change the logs.dir to something more local or a custom dir like this. log.dirs=/Users/xxx/yyy/software/confluent-5.3.1/mydata

Reboot your kafka cluster for the changes to take effect. Reboot your system and you will see the Topics are still present.

Answer 3

Go to kafka installation folder > config> server.properties

search for log.dirs in that file, change path from /tmp/logs to local directory. Restart kafka server and you will see topics created will be saved in that local folder we have changed in config file.