Using celery to build microservices

Question

I am going break down a project into small microservices.

All microservices are cron-based. I am thinking of celery as a task distribution as well a mechanism to run periodic tasks (celerybeat).

I don't want to build multiple celery app per microserverice as that will increase overhead of having multiple brokers and multiple flower system to use for monitoring.

I tried with single app on multiple servers but I failed. My needs with celery are :

1. I need to have independent servers for each microservice
2. Task belonging to certain microservice should execute only on their servers; no sharing of task among other servers
3. In case microservice is down i don't want celerybeat to clog the broker with thousands of pending tasks, resulting in halting service at other microservices.
4. In do not have any need of communication between microservices.

I tried separating queues per worker which doesn't seem to be possible I tried one worker per server but i need more than one worker on per microservices

Answer 1

For your use case, a simple queue based routing from a single broker should suffice.

Keep only 1 broker running in any one server or on a separate server.

Now while queuing up the tasks, add them to separate queues.

From micro service 1:

In [2]: add.apply_async(args=(12, 1), queue='queue1')
Out[2]: <AsyncResult: 2fa5ca61-47bc-4c2c-be04-e44cbce7680a>

Start a worker to consume only this queue

celery worker -A tasks -l info -Q queue1

From micro service 2:

In [2]: sub.apply_async(args=(12, 1), queue='queue2')
Out[3]: <AsyncResult: 4d42861c-737e-4b73-bfa8-6d1e86241d57>

Start a worker to consume only this queue

celery worker -A tasks -l info -Q queue2

This will make sure that tasks from a microservice will get executed by worker from that microservice only.