Celery - schedule periodic tasks starting at a specific time

Question

What is the best way to schedule a periodic task starting at specific datetime?

(I'm not using cron for this considering I've the need to schedule about a hundred remote rsyncs, where I compute the remote vs local offset and would need to rsync each path the second the logs are generated in each host.)

By my understanding the celery.task.schedules crontab class only allows specifying hour, minute, day of week. The most useful tip I've found so far was this answer by nosklo.

Is this the best solution? Am I using the wrong tool for the job?

Answer 1

Celery seems like a good solution for your scheduling problem: Celery's PeriodicTasks have run time resolution in seconds.

You're using an appropriate tool here, but the crontab entry is not what you want. You want to use python's datetime.timedelta object; the crontab scheduler in celery.schedules has only minute resolution, but using timedelta's to configure the PeriodicTask interval provides strictly more functionality, in this case, per second resolution.

e.g. from the Celery docs

>>> from celery.task import tasks, PeriodicTask
>>> from datetime import timedelta
>>> class EveryThirtySecondsTask(PeriodicTask):
...     run_every = timedelta(seconds=30)
...
...     def run(self, **kwargs):
...         logger = self.get_logger(**kwargs)
...         logger.info("Execute every 30 seconds")

http://ask.github.com/celery/reference/celery.task.base.html#celery.task.base.PeriodicTask

class datetime.timedelta(days=0, seconds=0, microseconds=0, milliseconds=0, minutes=0, hours=0, weeks=0)

The only challenge here is that you have to describe the frequency with which you want this task to run rather than at what clock time you want it to run; however, I would suggest you check out the Advanced Python Scheduler http://packages.python.org/APScheduler/

It looks like Advanced Python Scheduler could easily be used to launch normal (i.e. non Periodic) Celery tasks at any schedule of your choosing using it's own scheduling functionality.