celery worker not working though rabbitmq has queue buildup

Question

I am getting in touch with celery and I wrote a task by following Tutorial but somehow worker not getting up and I get following log After entering command:

celery worker -A tasks -l debug

I get a log:

Running a worker with superuser privileges when the
worker accepts messages serialized with pickle is a very bad idea!

If you really want to continue then you have to set the C_FORCE_ROOT
environment variable (but please think about this before you do).

User information: uid=0 euid=0 gid=0 egid=0

And here is my task:

from celery import Celery

app = Celery('tasks', backend='amqp',broker='amqp://sanjay:**@localhost:5672//')

@app.task
def gen_prime(x):
    multiples = []
    results = []
    for i in xrange(2, x+1):
        if i not in multiples:
            results.append(i)
            for j in xrange(i*i, x+1, i):
                multiples.append(j)
    return results

Though in rabbitmq admin console I see some queue build up when I try to generate prime numbers in ipython console but i am not getting result back on the console.

Here is my console action:

>>> from tasks import gen_prime
>>> pr=gen_prime.delay(10000)
>>> pr.ready()
False
>>> 
>>> pr.ready()
False
>>> pr.ready()
False

I am trying to solve this one from last 3 days but I was not able to solve it.

Answer 1

The error message pretty much tells you what's going on in this case. You're trying to run the worker as root (generally a bad idea due to security concerns). If you want to override this and allow it to run, you must set your environment:

export C_FORCE_ROOT="true"

Then run the worker.

Or you can just run it as a different user, which is preferred. You can search for how to add a user. Then you simply login as that user or su and execute your worker.

Since you tagged this digital ocean, here is a link to their tutorial on how to add a user:

https://www.digitalocean.com/community/tutorials/how-to-add-and-delete-users-on-ubuntu-12-04-and-centos-6

Also, celery has some docs regarding how to daemonize your workers. I usually use the supervisord method.

https://celery.readthedocs.org/en/latest/tutorials/daemonizing.html#centos

Answer 2

Don't run celery workers as root.

I recommend using supervisord to manage celery workers - you can use the user configuration directive to specify which user to run the celery workers as.