Sending Apache Kafka data on web page

Question

I am building a real time energy monitoring system, where the data are from sensors. There will be new data every second. The data used will be aggregated to be rendered as charts. I have looked into real time stream processing with big amounts of data, and it lead me to Apache Kafka.

Right now my web app is using Express js. I am using kafka-node library. Now currently, I manually insert new data through the command line as a producer. In my server code, I have set-up a consumer that listens to Topic1.

Server code:

var express = require('express');
var app = express();
var http = require('http').Server(app);
var bodyParser = require('body-parser');
var urlencodedParser = bodyParser.urlencoded({ extended: false });

var server = app.listen(3001, ()=>{
  console.log("app started on port 3001");
});

var io = require('socket.io').listen(server);


var kafka = require('kafka-node');

let Consumer = kafka.Consumer,
    client = new kafka.Client(),
    consumer = new Consumer(client,
      [
        {topic: 'Topic1', partition: 0}
      ],
      {
        autoCommit: false
      }
  );

app.use(express.static('public'));

consumer.on('message', (message) => {
  console.log(message.value);
  callSockets(io, message.value);
});

function callSockets(io, message){
  io.sockets.emit('update', message);
}

Client code:

<script type="text/javascript">
  var socket = io('http://localhost:3001');

  socket.on('connect', ()=>{
    console.log("connected");
  })

  socket.on('update', (data) =>{
    console.log(data);
  })
</script>

I am using socket.io to emit the message consumed in Kafka. Is there any other way to send Kafka data to client side? It just seems to me using socket.io is not too elegant here. Have I approached it the right way? Any recommendations welcome!

Thank you.

Answer 1

Without speaking to your solution specifically, I think you're doing the right thing creating a dedicated API for the client to read data from. The client needs to be able to receive updates from somewhere. The only "faster" way would be to allow the client to directly pull from Kafka which would increase risk substantially as Kafka is not designed to be publicly accessible.

So your solution with node.js is actually rather elegant compared to doing the same thing in C# or Java which would require a lot more code.