What kafka.common.OffsetOutOfRangeException means

Question

I'm trying to load data through Apache Kafka and constantly getting this error:

kafka.common.OffsetOutOfRangeException: offset 1003786 is out of range at kafka.log.Log$.findRange(Log.scala:46) at kafka.log.Log.read(Log.scala:264) at kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$readMessageSet(KafkaRequestHandlers.scala:112) at kafka.server.KafkaRequestHandlers$$anonfun$2.apply(KafkaRequestHandlers.scala:101) at kafka.server.KafkaRequestHandlers$$anonfun$2.apply(KafkaRequestHandlers.scala:100) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34) at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34) at scala.collection.TraversableLike$class.map(TraversableLike.scala:206) at scala.collection.mutable.ArrayOps.map(ArrayOps.scala:34) at kafka.server.KafkaRequestHandlers.handleMultiFetchRequest(KafkaRequestHandlers.scala:100) at kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$3.apply(KafkaRequestHandlers.scala:40) at kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$3.apply(KafkaRequestHandlers.scala:40) at kafka.network.Processor.handle(SocketServer.scala:296) at kafka.network.Processor.read(SocketServer.scala:319) at kafka.network.Processor.run(SocketServer.scala:214) at java.lang.Thread.run(Thread.java:724)

What does this exception mean and how I can fix it?

Answer 1

OffsetOutOfRangeException generally indicates that client has requested a range no longer available on the server.
This could happen as the topic log does not exists anymore based on the retention policy in your Kafka setup.
if you are using SimpleConsumer you will need to handle the OffsetOutOfRange Exception in your code. Ideally your consumer should issue an OffsetRequest to get the latest/earliest offset currently available on the server, and then use this in your FetchRequest (as a parameter)