Django-Celery in production?

Question

So I've been trying to figure out how to make scheduled tasks, I've found Celery and been able to to make simple scheduled tasks. To do this I need to open up a command line and run celery -A proj beat for the tasks to happen. This works fine in a development environment, but when putting this into production that will be an issue.

So how can I get celery to work without the command line use? When my production server is online, how can I make sure my scheduler goes up with it? Can Celery do this or do I need to go down another method?

Answer 1

We use Celery in our production environment, which happens to be on Heroku. We are in the process of moving to AWS. In both environments, Celery hums along nicely.

It would be helpful to understand what your production environment will look like. I'm slightly confused as to why you would be worried about turning off your computer, as using Django implies that you are running serving up a website... Are you serving your website from your laptop??

Anyway, assuming that you are going to run your production server from a cloud platform, all you have to do is send whatever command lines you need to run Django AND the command lines for Celery (as you have already noted in your question).

In terms of configuration, you say that you have 'scheduled' tasks, so that implies you have set up a beat schedule in your config.py file. If not, it should look something like this (assumes you have a module called tasks.py which holds your celery task definitions:

from celery.schedules import crontab

beat_schedule = {
    'task1': {
        'task': 'tasks.task_one',
        'schedule': 3600
    },
    'task2': {
        'task': 'tibController.tasks.update_old_retail',
        'schedule': crontab(hour=12, minute=0, day_of_week='mon-fri'
    }
}

Then in your tasks.py just call the config file you just do this:

from celery import Celery
import config

app = Celery('tasks')
app.config_from_object(config)

You can find more on crontab in the docs. You can also checkout this repo for a simple Celery example.

In summary:

1. Create a config file that identifies which tasks to run when
2. Load the config file into your Celery app
3. Get a cloud platform to run your code on.
4. Run celery exactly like you have already identified

Hope that helps.