Is it possible to integrate celery with Kafka

Question

I want to introduce multiprocessing in my code using celery. But currently my queue implementation is in Kafka.

Currently celery website mentions of only these 4 brokers: http://docs.celeryproject.org/en/master/getting-started/brokers/index.html#broker-overview

Is it possible to integrate Celery with Kafka something similar to RabbitMQ mentioned below:

from celery import Celery
app = Celery('tasks', broker='pyamqp://guest@localhost//')
@app.task
def add(x, y):
    return x + y

Answer 1

There was discussion on having kafka as broker for celery, And there is a partial implementation of it. I hope this helps you

https://github.com/celery/kombu/issues/301

Looks like it will be available this Oct 31 or so.

https://github.com/celery/kombu/milestone/2

Answer 2

As you are searching for multiprocessing solution for Kafka I would recommend to take a look at Faust library.

Here is an example (very close to yours) in comparison with Celery:

https://faust.readthedocs.io/en/latest/playbooks/vscelery.html

import faust

app = faust.App('myapp', broker='kafka://')

@app.agent()
async def add(stream):
    async for op in stream:
        yield op.x + op.y

Also one of the creators of Faust is the author of Celery.

I am telling that because at this moment already 7 years have passed since the creation of the ticket mentioned by @user3382968, and still without progress.