Where should you update Celery settings? On the remote worker or sender?

Question

Where should you update celery settings? On the remote worker or the sender?

For example, I have an API using Django and Celery. The API sends remote jobs to my remote workers via a broker (RabbitMQ). The workers are running a python script (not using Django) sometimes these works spawn sub tasks.

I've created celery settings on both sides (sender and worker) i.e. they both need the setting BROKER_URL. However, say I want to add the setting CELERY_ACKS_LATE = True, which end do I add this setting to? Each of the remote workers or the sender (API)?

Both the API and the remote workers connect to the same Broker, each start celery differently. The API creates a celery instance via Django __init__.py and the workers start celery via supervisor i.e. celery -A tasks worker -l info

Answer 1

the django celery settings affects only workers running on the django server itself.

if all your workers are remote workers (the way as i do it), then on the sender side all you need is to put the configuration necessary to submit a task to the task queue.

and all the other settings need to be set on the remote workers.

and for the tasks, on the sender side, all i need to do is to define the task signature like this:

@app.task(name='report_task')
def reportTask(self, link):
    pass

then on the worker side, you need to create a new Celery app with the same name and pointing to the same broker; for other celery settings you need to declare them on the remote workers.

and implement the tasks logic on the remote workeres (you can have different tasks logic per worker as long as they have the same task name and the function arguments)

Answer 2

CELERY_ACKS_LATE = True belongs to worker. It describes if worker should mark the task 'acknowledged' immediately after consuming (before completion) or after completing (late). Both methods have their drawbacks and I think you know what you're doing.

Of course it would be better to have single configuration file for both parties and use it. For example have the common codebase for entire project and after updating the file in VCS and deploy - restart all parties.

But in this case with this particular flag you can restart only workers.