Consume kafka messages from django app

Question

I'm designing a django based web application capable of serving via Web sockets data that need to be consumed from Kafka topic.

At this time, I came up with a solution splitted in two components: one component which consumes from kafka, perform some basic operations over retrieved data, and send the result to the django app using an http request. After request have been received, a message is written over a specific django channel.

Is there a better architecture to address this kind of scenario? Should I enclose all the Kafka part in a "while True" loop in a celery async task? Should I spawn a new process when django starts? If so, can I still use the django signals to send the data via web socket?

Thanks, Fb

Answer 1

yes, you can use your django code/repository and build separate app/program to deal with kafka queue and database through django ORM

just add at begin of this program code like

sys.path.append(os.getcwd())
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "<your_app>.settings")
django.setup()

and then you can use your models in this program, like

from <your_app>.models.timeslots import TimeSlotReserve

also good idea is to add some multithreading to this separate app