How to schedule a single-time event in Django in Heroku?

Question

I have a question about the software design necessary to schedule an event that is going to be triggered once in the future in Heroku's distributed environment.

I believe it's better to write what I want to achieve, but I have certainly done my research and could not figure it out myself even after two hours of work.

Let's say in my views.py I have a function:

def after_6_hours():
    print('6 hours passed.')


def create_game():
    print('Game created')
    # of course time will be error, but that's just an example
    scheduler.do(after_6_hours, time=now + 6)

so what I want to achieve is to be able to run after_6_hours function exactly 6 hours after create_game has been invoked. Now, as you can see, this function is defined out of the usual clock.py or task.py or etc etc files.

Now, how can I have my whole application running in Heroku, all the time, and be able to add this job into the queue of this imaginary-for-now-scheduler library?

On a side note, I can't use Temporizer add-on of Heroku. The combination of APScheduler and Python rq looked promising, but examples are trivial, all scheduled on the same file within clock.py, and I just simply don't know how to tie everything together with the setup I have. Thanks in advance!

Answer 1

In Heroku you can have your Django application running in a Web Dyno, which will be responsible to serve your application and also to schedule the tasks. For example (Please note that I did not test run the code):

Create after_hours.py, which will have the function you are going to schedule (note that we are going to use the same source code in worker too).

def after_6_hours():
        print('6 hours passed.')

in your views.py using rq (note that rq alone is not enough in your situation as you have to schedule the task) and rq-scheduler:

from redis import Redis
from rq_scheduler import Scheduler
from datetime import timedelta

from after_hours import after_6_hours

def create_game():
    print('Game created')

    scheduler = Scheduler(connection=Redis()) # Get a scheduler for the "default" queue
    scheduler.enqueue_in(timedelta(hours=6), after_6_hours) #schedules the job to run 6 hours later.

Calling create_game() should schedule after_6_hours() to run 6 hours later.
Hint: You can provision Redis in Heroku using Redis To Go add-on.

Next step is to run rqscheduler tool, which polls Redis every minute to see if there is any job to be executed at that time and places it in the queue(to which rq workers will be listening to).

Now, in a Worker Dyno create a file after_hours.py

def after_6_hours():
    print('6 hours passed.')
    #Better return something

And create another file worker.py:

import os

import redis
from rq import Worker, Queue, Connection

from after_hours import after_6_hours

listen = ['high', 'default', 'low'] # while scheduling the task in views.py we sent it to default

redis_url = os.getenv('REDISTOGO_URL', 'redis://localhost:6379')

conn = redis.from_url(redis_url)

if __name__ == '__main__':
    with Connection(conn):
        worker = Worker(map(Queue, listen))
        worker.work()

and run this worker.py

python worker.py

That should run the scheduled task(afer_6_hours in this case) in Worker Dyno. Please note that the key here is to make the same source code (after_hours.py in this case) available to worker too. The same is emphasized in rq docs

Make sure that the worker and the work generator share exactly the same source code.

If it helps, there is a hint in the docs to deal with different code bases.

For cases where the web process doesn't have access to the source code running in the worker (i.e. code base X invokes a delayed function from code base Y), you can pass the function as a string reference, too.

 q = Queue('low', connection=redis_conn)
 q.enqueue('my_package.my_module.my_func', 3, 4)

Hopefully rq-scheduler too respects this way of passing string instead of function object.

You can use any module/scheduling tool (Celery/RabbitMQ, APScheduler etc) as long as you understand this thing.