How to read json data using scala from kafka topic in apache spark

Question

I am new spark, Could you please let me know how to read json data using scala from kafka topic in apache spark.

Thanks.

Answer 1

The simplest method would be to make use of the DataFrame abstraction shipped with Spark.

val sqlContext = new SQLContext(sc)
val stream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](
                  ssc, kafkaParams, Set("myTopicName"))

stream.foreachRDD(
  rdd => {
     val dataFrame = sqlContext.read.json(rdd.map(_._2)) //converts json to DF
     //do your operations on this DF. You won't even require a model class.
        })