Can I ignore org.apache.kafka.common.errors.NotLeaderForPartitionExceptions?

Question

My Apache Kafka producer (0.9.0.1) intermittently throws a

org.apache.kafka.common.errors.NotLeaderForPartitionException

My code that performs the Kafka send resembles this

final Future<RecordMetadata> futureRecordMetadata = KAFKA_PRODUCER.send(new ProducerRecord<String, String>(kafkaTopic, UUID.randomUUID().toString(), jsonMessage));

try {
    futureRecordMetadata.get();
} catch (final InterruptedException interruptedException) {
    interruptedException.printStackTrace();
    throw new RuntimeException("sendKafkaMessage(): Failed due to InterruptedException(): " + sourceTableName + " " + interruptedException.getMessage());
} catch (final ExecutionException executionException) {
    executionException.printStackTrace();
    throw new RuntimeException("sendKafkaMessage(): Failed due to ExecutionException(): " + sourceTableName + " " + executionException.getMessage());
}

I catch NotLeaderForPartitionException within the catch (final ExecutionException executionException) {} block.

Is it OK to ignore this particular exception?

Has my Kafka message been sent successfully?

Answer 1

If you receive NotLeaderForPartitionException, your data was not written successfully.

Each topic partition is stored by one or multiple Brokers (with one leader; the remaining brokers are called followers) depending on your replication factor. A producer needs to send new messages to the leader Broker (data replication to followers happens internally).

Your producer client does not connect to the correct Broker, ie, to a follower instead of the leader (or to a broker that is not even a follower any longer), and this broker rejects your send request. This can happen if the leader changed but the producer still has outdated cached metadata about which broker is the leader for a partition.