Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Why would running scheduled tasks with Celery be preferable over crontab?

Considering Celery is already a part of the stack to run task queues (i.e. it is not being added just for running crons, that seems an overkill IMHO ).

How can its "periodic tasks" feature be beneficial as a replacement for crontab ? Specifically looking for following points.

  • Major pros/cons over crontab
  • Use cases where celery is better choice than crontab
  • Django specific use case: Celery vs crontab to run django based periodic tasks, when celery has been included in the stack as django-celery for queing django tasks.
like image 366
DhruvPathak Avatar asked Aug 12 '13 13:08

DhruvPathak


People also ask

What is crontab Celery?

Crontab schedulesIf you want more control over when the task is executed, for example, a particular time of day or day of the week, you can use the crontab schedule type: from celery.schedules import crontab app. conf.

How does Celery task queue work?

Dedicated worker processes constantly monitor task queues for new work to perform. Celery communicates via messages, usually using a broker to mediate between clients and workers. To initiate a task the client adds a message to the queue, the broker then delivers that message to a worker.

Is Celery a task queue?

Celery is an open source asynchronous task queue or job queue which is based on distributed message passing. While it supports scheduling, its focus is on operations in real time.


2 Answers

I've been using cron for a production website, and have switched to celery on a current project. I'm far more into celery than cron, here is why:

  • Celery + Celerybeat has finer granularity than cron. Cron cannot run more than once a minute, while celery can (I have a task run every 90 seconds which checks an email queue to send messages, and another which cleans the online users list).
  • A cron line has to call a script or a unique command, with absolute path and user info. Celery calls python functions, no need to write more than code.
  • With celery, to deploy to another machine, you generally just have to pull/copy your code, which is generally in one place. Deploying with cron would need more work (you can automate it but...)
  • I really find celery better suited than cron for routine cleaning (cache, database), and in general, for short tasks. Dumping a database is more a work for cron, however, because you don't want clutter the event queue with too long tasks.
  • Not the least, Celery is easily distributed across machines.
like image 162
Steve K Avatar answered Sep 28 '22 09:09

Steve K


Celery is indicated any time you need to coordinate jobs across multiple machines, ensure jobs run even as machines are added or dropped from a workgroup, have the ability to set expiration times for jobs, define multi-step jobs with graph-style rather than linear dependency flow, or have a single repository of scheduling logic that operates the same across multiple operating systems and versions.

like image 42
Chris Johnson Avatar answered Sep 28 '22 08:09

Chris Johnson