Dynamically Changing Celery Beat Settings

Question

Is there a way to tell celerybeat to change settings for a specific task while it is running?

The utility of this is best illustrated in this example:

I have a periodic task that checks a value every 30 seconds. Sometimes, based on an external trigger (that I can't predict) I will want this task to increase the polling frequency to 10s - for a few minutes.

Is this doable in a manageable way? I know I can change the task config and reload celery but that seems a messy way to do things...

Answer 1

On the schedules.py module you will find this:

class schedule(object):

    def is_due(self, last_run_at):
        """Returns tuple of two items `(is_due, next_time_to_run)`,
        where next time to run is in seconds.

        e.g.

        * `(True, 20)`, means the task should be run now, and the next
            time to run is in 20 seconds.

        * `(False, 12)`, means the task should be run in 12 seconds.

        You can override this to decide the interval at runtime,
        but keep in mind the value of :setting:`CELERYBEAT_MAX_LOOP_INTERVAL`,
        which decides the maximum number of seconds celerybeat can sleep
        between re-checking the periodic task intervals.  So if you
        dynamically change the next run at value, and the max interval is
        set to 5 minutes, it will take 5 minutes for the change to take
        effect, so you may consider lowering the value of
        :setting:`CELERYBEAT_MAX_LOOP_INTERVAL` if responsiveness is of
        importance to you.

        .. admonition:: Scheduler max interval variance

        The default max loop interval may vary for different schedulers.
        For the default scheduler the value is 5 minutes, but for e.g.
        the django-celery database scheduler the value is 5 seconds.

        """
        last_run_at = self.maybe_make_aware(last_run_at)
        rem_delta = self.remaining_estimate(last_run_at)
        rem = timedelta_seconds(rem_delta)
        if rem == 0:
            return True, self.seconds
        return False, rem

So, you can override the is_due method to set your own timedelta.