How to integrate Django with Kafka using Python?

Question

I want to pass input from Django to Kafka. I have fields a,b,c,d and h1,h2 and h3. All fields are input from the user. I want to start my Kafka process as soon as the user enters submit button on my Django front-end interface. All fields are being saved in Mysql.

I can only find possible solutions in Java using Mysql connectors.

Is there any method or connector with which i can pass my input from user on Django interface to kafka using python ?

a,b,c,d - Access Tokens

h1,h2,h3 - Search queries(keywords)

Answer 1

First install kafka-python, then import following statements in views.py.

1. from kafka import KafkaProducer
2. from kafka import KafkaConsumer
3. from json import loads
4. import json
5. import pickle //pickle converts data into byte array

Then write Producer view as following.

This code converts data into a byte array and send to kafka. Instead of sending json message v you can send your data like v=your data.

def kfk(request):   
    producer = KafkaProducer(bootstrap_servers='127.0.0.1:9092')
    v = {
        'msg': {
            'hello': 'world',
        },
    }
    serialized_data = pickle.dumps(v, pickle.HIGHEST_PROTOCOL)
    producer.send('Ptopic', serialized_data)
    return HttpResponse(200)

To consume data:

def cons(request):
    consumer = KafkaConsumer('Ptopic', 
        bootstrap_servers=['localhost:9092'], 
        api_version=(0, 10) 
        #,consumer_timeout_ms=1000
    )

    for message in consumer:
        deserialized_data = pickle.loads(message.value) 
        print(deserialized_data)

Note: Kafka consumer view should always in running mode then try to produce here Ptopic is my topic name.

Answer 2

Yes, just use the KafkaProducer from the kafka-python package and you will be set.

pip install kafka-python

Then in your Django function:

def myfunc(request):
  from kafka import KafkaProducer
  producer = KafkaProducer(bootstrap_servers='kafkaBroker:9092')
  producer.send('foobar', b'test')
  return HttpResponse(200)