Django Background tasks vs Celery

Question

I am trying to do some tasks in django that consume alot of time. For that, I will be running background tasks.

After some R&D, i have found two solutions:

• Celery with RabbitMQ.

• Django Background tasks.

Both options seem to fulfill the criteria but setting up Celery will require some work. Now as far as the second option is concerned, setup is fairly simple and in fairly quick amount of time, i can go on writing background tasks. Now my questions if i adopt the 2nd option is this:

1. How well does Django Background tasks perform ? (Scalability wise in Production environment).
2. Can i poll the tasks (after some time) in DB to check the task's status?
3. Architecture of Django-Background-tasks? Couldn't find any clear explanation about it's architecture (Or have I missed some resource?)
4. Again coming to the first point, how well does Django Background tasks perform in production. (Talking about prior experience of using this in prod.)

Answer 1

Setting up celery takes work (although less when using Redis). It's also serious tool with almost a decade of investment and widespread industry adoption.

As for performance, scaling behaviors of task systems which are backed by queues vs those backed by RDBMs are well understood – but may not be relevant to you as "scalability" is a very subjective term. This thread provides some good framing on the subject and questions.

Comparing stars on GitHub (bg tasks' 3XX vs Celery's 13XXX), you should realize Django-Background-tasks has a smaller user base, and you're probably going to need to get into the internals to understand the architecture and precise mechanics. That shouldn't stop you – just be prepared to DIY when answers aren't forthcoming.