How to automatically start Kafka upon system startup in Ubuntu?

Question

Does Kafka have an official way (e.g. an init.d script) to start Kafka on system start up?

The only official method to start Kafka I have seen is:

nohup ~/kafka/bin/kafka-server-start.sh ~/kafka/config/server.properties > ~/kafka/kafka.log 2>&1 &

I have tried to use a @reboot task in crontab -e however it did not start Kafka. Some people have also written custom init.d

There are also custom init.d scripts available (e.g. one, two, three) however they are all different and I am not familiar enough with init.d to understand which one, if any to implement.

How to start Kafka on system startup?

Answer 1

Here's how I configure Kafka to start automatically on Ubuntu 14.04:

sudo su
cp -R ~/kafka_2.11-0.10.0.1 /opt
ln -s /opt/kafka_2.11-0.10.0.1 /opt/kafka

Copy the following init script to /etc/init.d/kafka:

DAEMON_PATH=/opt/kafka/
PATH=$PATH:$DAEMON_PATH/bin

# See how we were called.
case "$1" in
  start)
        # Start daemon.
        echo "Starting Zookeeper";
        nohup $DAEMON_PATH/bin/zookeeper-server-start.sh -daemon /$DAEMON_PATH/config/zookeeper.properties 2> /dev/null && \
        echo "Starting Kafka";
        nohup $DAEMON_PATH/bin/kafka-server-start.sh -daemon /$DAEMON_PATH/config/server.properties 2> /dev/null
        ;;
  stop)
        # Stop daemons.
        echo "Shutting down Zookeeper";
        pid=`ps ax | grep -i 'org.apache.zookeeper.server' | grep -v grep | awk '{print $1}'`
        if [ -n "$pid" ]
          then
          kill -9 $pid
        else
          echo "Zookeeper was not Running"
        fi
        echo "Shutting down Kafka";
        pid=`ps ax | grep -i 'kafka.Kafka' | grep -v grep | awk '{print $1}'`
        if [ -n "$pid" ]
          then
          kill -9 $pid
        else
          echo "Kafka was not Running"
        fi
        ;;
  restart)
        $0 stop
        sleep 2
        $0 start
        ;;
  status)
        pid=`ps ax | grep -i 'org.apache.zookeeper.server' | grep -v grep | awk '{print $1}'`
        if [ -n "$pid" ]
          then
          echo "Zookeeper is Running as PID: $pid"
        else
          echo "Zookeeper is not Running"
        fi
        pid=`ps ax | grep -i 'kafka.Kafka' | grep -v grep | awk '{print $1}'`
        if [ -n "$pid" ]
          then
          echo "Kafka is Running as PID: $pid"
        else
          echo "Kafka is not Running"
        fi
        ;;
  *)
        echo "Usage: $0 {start|stop|restart|status}"
        exit 1
esac

exit 0

Make the kafka service with these commands:

chmod 755 /etc/init.d/kafka
update-rc.d kafka defaults

Now you should be able to start and stop the kafka service like this:

sudo service kafka start
sudo service kafka status
sudo service kafka stop

If you want to remove the Kafka service later, run update-rc.d -f kafka remove.

Answer 2

Download Kafka

cd /opt
sudo wget http://mirror.hosting90.cz/apache/kafka/2.5.0/kafka-2.5.0-src.tgz
sudo tar -zxvf kafka-2.5.0-src.tgz
sudo mv kafka-2.5.0-src kafka
sudo rm kafka-2.5.0-src.tgz
cd kafka
sudo ./gradlew jar -PscalaVersion=2.11.12

Install Zookeeper

sudo vi /etc/systemd/system/zookeeper.service

Edit zookeeper.service

[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
User=root
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh opt/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Start Zookeeper

sudo systemctl enable zookeeper.service

sudo systemctl start zookeeper.service

sudo systemctl status zookeeper.service

active (running)

Install Kafka

sudo vi /etc/systemd/system/kafka.service

Edit kafka.service

[Unit]
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
User=root
ExecStart=/bin/sh -c '/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties > /opt/kafka/kafka.log 2>&1'
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Start Kafka

sudo systemctl enable kafka.service

sudo systemctl start kafka.service

sudo systemctl status kafka.service

active (running)

Test Kafka works

create topic

sudo bin/kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic test-topic

put messages to topic

sudo bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test-topic  
> test message1
> test messate2
^C

read messages from topic

sudo bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic test-topic
test message1
test messate2
^C