How to properly manage rabbitmq with supervisord

Question

The current section in my supervisord.conf looks like:

[program:rabbitmq] command=/usr/sbin/rabbitmq-server

When I try to stop the rabbitmq with supervisord ( supervisorctl stop rabbitmq), the rabbitmq processes simply do not shut down. The rabbitmq documentation also mentions to never use kill but rather use rabbitmqctl stop . I'm guessing supervisord simply kills the processes - hence the poor results with rabbitmq. I couldn't find any options in supervisord to specify a custom stop command.

Do you have any recommendations?

Answer 1

My solution is to write a wrapper script named rabbitmq.sh as follows:

# call "rabbitmqctl stop" when exiting
trap "{ echo Stopping rabbitmq; rabbitmqctl stop; exit 0; }" EXIT

echo Starting rabbitmq
rabbitmq-server

After that, modify supervisord.conf:

[program:rabbitmq]
command=path/to/rabbitmq.sh 

Answer 2

You have answered your own question. In normal operation, never use kill on any process unless that is the documented normal way of managing it. In the case of RabbitMQ, the documented process is to use rabbitmqctl stop or to use rabbitmqserver stop.

There is no good reason to manage RabbitMQ with anything more sophisticated than a shell script that makes one attempt to restart via rabbitmqserver start. If that doesn't work right away, then RabbitMQ is down hard due to something like, lack of RAM, ran out of disk space, or a rogue system management tool deleted some of the rabbitmq binary components.

In normal operation RabbitMQ has an internal supervisor that will attempt to shutdown and restart RabbitMQ, so if you delete binaries, it will fail to restart. When using tools like chef, puppet, cfengine, don't repeatedly push out binary package files. Just check that everything is there as it should be.

Answer 3

This script starts RabbitMQ as a background process (using '&') which causes a pid file to be updated/created (see 'wait' under http://www.rabbitmq.com/man/rabbitmqctl.1.man.html).

After rabbit has been started, a loop is used to verify that the pid is still running. If rabbit crashes or is manually shut down (outside of supervisord) then the script will exit with 1 and supervisord takes over.

The echo >> ./rmq.txt file is there for debugging purposes and can be commented out in production (I used this to monitor the startup/shutdown/died status).

supervisord is happy because it can see a running process and an EXIT will trigger the stop_rmq function which calls 'rabbitmqctl stop' for a clean shutdown.

#!/bin/bash

# Script to manage RMQ with supervisord

# Shut down rmq
function stop_rmq {

  echo "Stopping RabbitMQ..."
  echo "Stopping RabbitMQ..." >> ./rmq.txt
  rabbitmqctl stop
  echo "RabbitMQ stopped"
  echo "RabbitMQ stopped" >> ./rmq.txt
  #exit 0
}

# Set up the trap
#trap stop_rabbit TERM KILL HUP INT SIGTERM SIGKILL SIGHUP SIGINT
trap stop_rmq exit

# Start rmq
echo "Starting RabbitMQ..."
echo "Starting RabbitMQ..." >> ./rmq.txt
# Start Rabbitmq in the background (causes the pid file to be updated)
# Note that the pid file location can be overridden with the rmq 'RABBITMQ_PID_FILE' variable
/usr/sbin/rabbitmq-server &
rabbitmqctl wait /var/lib/rabbitmq/mnesia/rabbit@$HOSTNAME.pid
echo "RabbitMQ Started"
echo "RabbitMQ Started" >> ./rmq.txt

while true; do
  #ps $(cat /var/lib/rabbitmq/mnesia/rabbit@$HOSTNAME.pid)
  ps -o pid,cmd,etime $(cat /var/lib/rabbitmq/mnesia/rabbit@$HOSTNAME.pid)
  if (($? > 0)); then
    echo "RabbitMQ Died"
    echo "RabbitMQ Died" >> ./rmq.txt
    exit 1
  fi
  #echo "Sleeping..."
  sleep 10
done

Here's the output generated by the script to supervisord:

foo@bar:/# supervisorctl tail rmq

Starting RabbitMQ...
Waiting for rabbit@a2d2c8f9cad2 ...
pid is 45220 ...

              RabbitMQ 3.3.5. Copyright (C) 2007-2014 GoPivotal, Inc.
  ##  ##      Licensed under the MPL.  See http://www.rabbitmq.com/
  ##  ##
  ##########  Logs: /var/log/rabbitmq/[email protected]
  ######  ##        /var/log/rabbitmq/[email protected]
  ##########
              Starting broker... completed with 0 plugins.
...done.
RabbitMQ Started
  PID CMD                             ELAPSED
45220 /usr/lib/erlang/erts-6.1/bi       00:05
  PID CMD                             ELAPSED
45220 /usr/lib/erlang/erts-6.1/bi       00:15
  PID CMD                             ELAPSED
45220 /usr/lib/erlang/erts-6.1/bi       00:25
  PID CMD                             ELAPSED
45220 /usr/lib/erlang/erts-6.1/bi       00:35
  PID CMD                             ELAPSED
45220 /usr/lib/erlang/erts-6.1/bi       00:45
  PID CMD                             ELAPSED
45220 /usr/lib/erlang/erts-6.1/bi       00:55
  PID CMD                             ELAPSED
45220 /usr/lib/erlang/erts-6.1/bi       01:05
  PID CMD                             ELAPSED
45220 /usr/lib/erlang/erts-6.1/bi       01:15
  PID CMD                             ELAPSED
45220 /usr/lib/erlang/erts-6.1/bi       01:25

Answer 4

I would advise you to use Monit (http://mmonit.com/), it is better suited for daemons such as RabbitMQ and it is also feature rich.

First of all, you must install the Monit package. If you are under Ubuntu/Debian:

sudo apt-get update
sudo apt-get install monit

Afterwards, you must create a configuration script. Here is a sample script to get you running (place it on /etc/monit/conf.d/):

set daemon 1800 
set logfile /var/log/monit.log

check process rabbit with pidfile /var/run/rabbitmq/pid
    start program = "/etc/init.d/rabbitmq-server start"
    stop program  = "/etc/init.d/rabbitmq-server stop"
    noalert foo@bar

Then, just restarts monit and you are finished:

 sudo /etc/init.d/monit restart