Manage RabbitMQ consumers

Question

I am planning to write a log processing application using RabbitMQ, Symfony2 and the RabbitMqBundle. The tool I am working on has to be highly available and must process millions of entries per day, so it's important that the consumers are always up and running (short breaks are fine), otherwise my queue might overflow after a while.

Are there best practices on how to manage the consumers (written in PHP), start/restart them in case of an error etc?

Thanks

Answer 1

I use this bash script to make sure that there are all required consumers running on imagepush.to:

#!/bin/bash

NB_TASKS=1
SYMFONY_ENV="prod"

TEXT[0]="app/console rabbitmq:consumer primary"
TEXT[1]="app/console rabbitmq:consumer secondary"

for text in "${TEXT[@]}"
do

NB_LAUNCHED=$(ps ax | grep "$text" | grep -v grep | wc -l)

TASK="/usr/bin/env php ${text} --env=${SYMFONY_ENV}"

for (( i=${NB_LAUNCHED}; i<${NB_TASKS}; i++ ))
do
  echo "$(date +%c) - Launching a new consumer"
  nohup $TASK &
done

done

If I remember correctly I took base from KnpBundles code.