Celery tasks disappear

Question

I have a django project running with cron script, executing a management command. This command creates in for cycle tasks for celery:

for r in pr:
    log_task(tasks_logger.info, "to_queue", r)
    remind.delay(r, now, send_all)

And the task looks like this:

class RTask(Task):
    abstract = True
    def on_failure(self, exc, task_id, args, kwargs, einfo):
        r = args[0]
        log_task(logger.error, exc, r)
        log_task(logger_tb.error, einfo, r)


@task(base=RTask)
def remind(r, now, send_all):
    log_task(logger.info, "from_queue", r)
    ....

As u can see, I have a logger before task execution and on first line inside it. The problem is - after the update of the project code (another programmer added other tasks and celery version update) majority of my tasks start vanishing. My log file looks like this (only 1 of 8-10 tasks executed):

[2014-03-12 12:45:08,806]  106152122   INFO    to_queue
[2014-03-12 12:45:08,819]  106138932   INFO    to_queue
[2014-03-12 12:45:08,915]  106121944   INFO    to_queue
[2014-03-12 12:45:08,916]  110418819   INFO    from_queue
[2014-03-12 12:45:08,922]  106075777   INFO    to_queue

The celery log file don't contains any helpful info. So does rabbit. It has lots of this stuff, but its not connected with my tasks, or does it?

[2014-03-12 12:58:43,091: INFO/MainProcess] Got task from broker: celery.chord_unlock[7fe8f29f-69e1-456c-8a14-7fae0cfacc33] eta:[2014-03-12 12:58:44.089401+00:00]
[2014-03-12 12:58:43,092: INFO/MainProcess] Task celery.chord_unlock[7fe8f29f-69e1-456c-8a14-7fae0cfacc33] retry: Retry in 1s
[2014-03-12 12:58:43,092: INFO/MainProcess] Task celery.chord_unlock[7b1d4a6b-9a34-43e9-98c9-851c93ace5ce] retry: Retry in 1s

What could be possibly the problem? How can I trace task to understand when it disappears?

Please help =)

Answer 1

It's possible that you have celery processes running in the background, relics of previous launches that weren't shut down properly, that might be consuming the messages. try to see if you have such workers by running

ps aux | grep celery

in the command line. The following command will automatically kill all such orphan celery workers for you:

ps aux | grep celery | awk '{system("kill -9 " $2)}'

I execute it before launching my app

Answer 2

Here is the reason of my problem:
http://docs.python.org/2/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes
Although logging is thread-safe, and logging to a single file from multiple threads in a single process is supported, logging to a single file from multiple processes is not supported, because there is no standard way to serialize access to a single file across multiple processes in Python. If you need to log to a single file from multiple processes, one way of doing this is to have all the processes log to a SocketHandler, and have a separate process which implements a socket server which reads from the socket and logs to file. (If you prefer, you can dedicate one thread in one of the existing processes to perform this function.) This section documents this approach in more detail and includes a working socket receiver which can be used as a starting point for you to adapt in your own applications.
http://docs.python.org/2/howto/logging-cookbook.html#sending-and-receiving-logging-events-across-a-network
When a had small load everything worked perfect, but with its increase I faced the problem.