Scheduled Celery tasks on RabbitMQ remain unacknowledged past their specified run time

Question

Having trouble getting scheduled tasks to run at a specified future time while using Celery and RabbitMQ.

Using Django on a Heroku server, with the RabbitMQ add-on.

The Problem:

Sometimes the tasks don't run at all, sometimes they do run, but the times that they run at are off by a significant margin (like an hour).

Example task that did not run:

When I try to run a task with a countdown or ETA, it never actually executes. This is an example ETA task that did not run:

>>> dummy_task.apply_async(eta=datetime.datetime.now() + timedelta(seconds=60))
<AsyncResult: 03001c1c-329e-46a3-8180-b115688e1865>

Resulting Log:

2012-07-24T14:03:08+00:00 app[scheduler.1]: [2012-07-24 10:03:08,909: INFO/MainProcess]
    Got task from broker: events.tasks.dummy_task[910ff406-d51c-4c29-bdd1-fec1a8168c12]     
    eta:[2012-07-24 10:04:08.819528+00:00]

One minute later nothing happens. The unacknowledged message count in my Heroku RabbitMQ management console increases by one and stays there.

This works:

I've made sure that the celery task is properly registered and RabbitMQ is configured to accept tasks by verifying that I can run the task using the delay() method.

>>> dummy_task.delay()
<AsyncResult: 1285ff04-bccc-46d9-9801-8bc9746abd1c>

Resulting Log:

2012-07-24T14:29:26+00:00 app[worker.1]: [2012-07-24 10:29:26,513: INFO/MainProcess] 
    Got task from broker: events.tasks.dummy_task[1285ff04-bccc-46d9-9801-8bc9746abd1c]

....

2012-07-24T14:29:26+00:00 app[worker.1]: [2012-07-24 10:29:26,571: INFO/MainProcess] 
    Task events.tasks.dummy_task[1285ff04-bccc-46d9-9801-8bc9746abd1c] 
    succeeded in 0.0261888504028s: None

Any help on this would be greatly appreciated. Thanks a lot!

Answer 1

I sort of figured it out. Because I was debugging, I was naturally setting tasks to run in 5 or 10 minutes. I don't know exactly why, but my set up has a hard time handling this. It's like it needs a certain amount of time to process and queue the task. Anyway, tasks scheduled for more than an hour or so in advance have been running correctly.