Celery Beat on Amazon ECS

Question

I am using Amazon Web Services ECS (Elastic Container Service). My task definition contains Application + Redis + Celery and these containers are defined in task definition. Automatic scaling is set, so at the moment there are three instances with same mirrored infrastructure. However, there is a demand for a Celery Beat instance for scheduled tasks, so Celery Beat would be a great tool, since Celery is already in my infrastructure.

But here is the problem: if I add Celery Beat container together with other containers (add it to task definition), it will be mirrored and multiple instances will execute same scheduled tasks at the same moment. What would be a solution to this infrastructure problem? Should I create a seperate service?

Answer 1

We use single-beat to solve this problem and it works like a charm:

Single-beat is a nice little application that ensures only one instance of your process runs across your servers.

Such as celerybeat (or some kind of daily mail sender, orphan file cleaner etc...) needs to be running only on one server, but if that server gets down, well, you go and start it at another server etc.

You should still set the number of desired tasks for the service to 1.

Answer 2

You can use ECS Task Placement strategy to place your Celery Beat task and choose "One Task Per Host". Make sure to choose Desire state to "1". In this way, your celery beat task will run only in 1 container in your cluster.

Ref: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_run_task.html

The desired task is the number of tasks you want to run in the cluster. You may set the "Number of tasks" while configuring the service or in the run task section. You may refer the below links for references.

Configuring service:

Ref: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-service.html

Run Task:

Ref: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_run_task.html

Let me know if you find any issue with it.