Using celery with already existing MySQL table instead of a broker , is it possible?

Question

Task at hand :

There is a MySQL table , where a user activity is pushed as a row. That activity needs to be processed later.

id    |  activity_type  |  activity_data | creation_time | status
23          EMAIL          {....... }       2013-02-01     UNPROCESSED

Processing without Celery :

• Use a script which picks up N records to process, processes them and then updates the status as PROCESSED for processed records.

How can the same be achieved using Celery, I would like to use the same functionality to mark tasks instead of using a broker ? i.e. task add -> take 10 oldest rows added to the mysql table with status UNPROCESSED , which has the task related data as well.

on task finish -> mark the rows as PROCESSED.

How can celery be told that instead of pushing the task onto the broker, it has to retrieve tasks from the MySQL table ?

I am a beginner at Celery, hence not aware of all its functionalities. Using MySQL as broker is not recommended,but I would like to know the feasibility.

Answer 1

I think you're misunderstanding the way celery works. You can't just replace its broker with a MySQL table of your own design - well, not without making substantial changes to its source code.

The broker is an internal part of celery which it uses to keep track of its tasks, using its own internal format, so there's no inherent advantage to using a MySQL broker solely because that's the way in which you currently store the information necessary to perform your tasks.

You can still use celery if you like, but you'd have to write the code necessary to translate your user activity table into celery tasks.

However, I'd recommend experimenting with celery first to...

1. get a basic understanding of how to use it
2. determine if it's a good solution for what you're trying to achieve

Start off with the celery tutorial, and see how it goes from there.