When does kafka change leader?

Question

I was running my services that work with kafka already for a year and no spontaneous changes of leader happens. But for the last 2 weeks that started happens quite often. Kafka log on that:

• [2015-09-27 15:35:14,826] INFO [ReplicaFetcherManager on broker 2] Removed fetcher for partitions [myTopic] (kafka.server.ReplicaFetcherManager)
• [2015-09-27 15:35:14,830] INFO Truncating log myTopic-0 to offset 11520979. (kafka.log.Log)
• [2015-09-27 15:35:14,845] WARN [Replica Manager on Broker 2]: Fetch request with correlation id 713276 from client ReplicaFetcherThread-0-2 on partition [myTopic,0] failed due to Leader not local for partition [myTopic,0] on broker 2 (kafka.server.ReplicaManager)
• [2015-09-27 15:35:14,857] WARN [Replica Manager on Broker 2]: Fetch request with correlation id 256685 from client mirrormaker-1 on partition [myTopic,0] failed due to Leader not local for partition [myTopic,0] on broker 2 (kafka.server.ReplicaManager)
• [2015-09-27 15:35:20,171] INFO [ReplicaFetcherManager on broker 2] Removed fetcher for partitions [myTopic,0] (kafka.server.ReplicaFetcherManager)

What can cause switching leader? If there is info in some kafka documentation - please - just point the link. I've failed to find.

---

System configuration

kafka version: kafka_2.10-0.8.2.1

os: Red Hat Enterprise Linux Server release 6.5 (Santiago)

server.properties (differs from default):

• broker.id=001
• socket.send.buffer.bytes=1048576
• socket.receive.buffer.bytes=1048576
• socket.request.max.bytes=104857600
• log.flush.interval.messages=10000
• log.flush.interval.ms=1000
• log.retention.bytes=-1
• controlled.shutdown.enable=true
• auto.create.topics.enable=false

Answer 1

It appears like lead broker is down for that partition. It might be that data directroy(log.dirs) configured in server.properties is out of space and broker is not able to accommodate. Also, what is replication factor of topic and cluster size of brokers?