Django and Celery - How to Distribute?

Question

I'm trying to distribute Django and Celery.

I've created a small project with Django and Celery. Django will request a Celery Worker to work on some data on the database. Then the data is passed back to Django.

My idea is that:

1. Django stack installed on one server
2. Message queue (RabbitMQ) on one server
3. Celery worker on one server

Hence 3 Servers in Total

However, the problem is celery has to use some code from Django, for example models, because it accesses the model. Hence, it would also require settings.py file to know what are the servers.

Does this mean that for #3, I would need to install Django and Celery on the server, but disable Django and only run celery? For example celery -A PROJECT_NAME worker -l INFO, but without an Apache Server for Django?

Answer 1

If you want your celery workers to operate on a different server, you need to make sure that all the resources required by the worker are accessible from that server.

For example, if you have a simple task, you can copy only the code required for that task to the server. If your worker needs any other resources like some other code, files, db you need to make sure it has access.

Really, if you want to have two servers working on the same tasks, you will have to use a simple web interface (such as Flask) to communicate between the servers (and extend the functionality of your queue). Then, you will have to ensure they are both using the same data source.

Consider hosting your database remotely, or have the remote server access the database remotely. Either way, any workers running on a server will need access to the database and all source code necessary to complete the task. Then, you must simply have the two servers share a messaging queue.

Source: how to configure and run celery worker on remote system