How to give name to each node in celery

Question

We are using celery with our flask application. There are many queues for our various tasks and we use supervisord to run queues. We use cloudamqp broker.

Example supervisor configuration is like below:

[program:my-queue]
command=/home/ubuntu/opt/proect/venv/bin/celery -A async_runner worker -Q my_queue --loglevel=INFO --without-gossip --without-mingle --autoscale=1,1  -c 1
environment=PYTHONPATH=/home/ubuntu/opt/project/,PRODUCTION_ENVIRONMENT=true
directory=/home/ubuntu/opt/project/app

process_name = %(program_name)s_%(process_num)02d
user=ubuntu
numprocs=2
autostart=true
autorestart=true
startsecs=10
stopwaitsecs = 600
priority=998

We get following error and queues just stop.

[2017-04-06 12:43:06,759: WARNING/MainProcess] /home/ubuntu/opt/project/venv/local/lib/python2.7/site-packages/kombu/pidbox.py:75: UserWarning: A node named celery@ip-xxx-yy-yy-yy is already using this process mailbox!

Maybe you forgot to shutdown the other node or did not do so properly?
Or if you meant to start multiple nodes on the same host please make sure
you give each node a unique node name!

  warnings.warn(W_PIDBOX_IN_USE.format(node=self))

Question: How to give names to each node?

When I run celery -A async_runner status it gives 5 nodes online message.

celery@ip-ip_value_here-: OK
celery@ip-ip_value_here-: OK
celery@ip-ip_value_here-: OK
celery@ip-ip_value_here-: OK
celery@ip-ip_value_here-: OK

5 nodes online.

Answer 1

Here is how we run one celery worker for each queue in our company. The entry point contains this:

echo QUEUES: ${QUEUES}  # comma separated list of queue names
export NUM_QUEUES=$(python -c "print len('$QUEUES'.split(','))")
echo NUM_QUEUES: ${NUM_QUEUES}

supervisord &

The supervisor config file has this:

[program:worker]
command=/path/to/celery_worker.sh %(process_num)s
# supervisor uses the special ENV_ prefix to get the environment variable set above
numprocs=%(ENV_NUM_QUEUES)s

And finally celery_worker.sh contains something like:

QUEUE=$(python -c "print '$QUEUES'.split(',')[$1]")  # $1 = process_num above
celery worker -n "worker.${QUEUE}" -Q ${QUEUE}  # -n sets the name of the worker