Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Kafka - Broker fails because all log dirs have failed

Tags:

I am attempting a simple kafka config on Windows os. My zookeeper and kafka installations use default configs (except for data and log dir paths). I can start kafka and produce/consume messages without issue; however, when the broker attempts to delete old messages (I set log retention to 100 ms), I get the following error:

    java.nio.file.FileSystemException: C:\Workspace\kafka_2.11-1.1.0\kafka-logs\discrete-topic-0\00000000000000000000.log -> C:\Workspace\kafka_2.11-1.1.0\kafka-logs\discrete-topic-0\00000000000000000000.log.deleted: The process cannot access the file because it is being used by another process.          at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)         at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)         at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)         at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)         at java.nio.file.Files.move(Files.java:1395)         at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:697)         at org.apache.kafka.common.record.FileRecords.renameTo(FileRecords.java:212)         at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:415)         at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:1601)         at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:1588)         at kafka.log.Log$$anonfun$deleteSegments$1$$anonfun$apply$mcI$sp$1.apply(Log.scala:1170)         at kafka.log.Log$$anonfun$deleteSegments$1$$anonfun$apply$mcI$sp$1.apply(Log.scala:1170)         at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)         at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)         at kafka.log.Log$$anonfun$deleteSegments$1.apply$mcI$sp(Log.scala:1170)         at kafka.log.Log$$anonfun$deleteSegments$1.apply(Log.scala:1161)         at kafka.log.Log$$anonfun$deleteSegments$1.apply(Log.scala:1161)         at kafka.log.Log.maybeHandleIOException(Log.scala:1678)         at kafka.log.Log.deleteSegments(Log.scala:1161)         at kafka.log.Log.deleteOldSegments(Log.scala:1156)         at kafka.log.Log.deleteRetentionMsBreachedSegments(Log.scala:1228)         at kafka.log.Log.deleteOldSegments(Log.scala:1222)         at kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:854)         at kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:852)         at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)         at scala.collection.immutable.List.foreach(List.scala:392)         at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)         at kafka.log.LogManager.cleanupLogs(LogManager.scala:852)         at kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:385)         at kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:110)         at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:62)         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)         Suppressed: java.nio.file.FileSystemException: C:\Workspace\kafka_2.11-1.1.0\kafka-logs\discrete-topic-0\00000000000000000000.log -> C:\Workspace\kafka_2.11-1.1.0\kafka-logs\discrete-topic-0\00000000000000000000.log.deleted: The process cannot access the file because it is being used by another process.                  at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)                 at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)                 at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)                 at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)                 at java.nio.file.Files.move(Files.java:1395)                 at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:694)                 ... 32 more  And then...  [2018-08-01 18:14:01,479] INFO [ReplicaManager broker=0] Stopping serving replicas in dir C:\Workspace\kafka_2.11-1.1.0\kafka-logs (kafka.server.ReplicaManager) [2018-08-01 18:14:01,480] ERROR Uncaught exception in scheduled task 'kafka-log-retention' (kafka.utils.KafkaScheduler) org.apache.kafka.common.errors.KafkaStorageException: Error while deleting segments for discrete-topic-0 in dir C:\Workspace\kafka_2.11-1.1.0\kafka-logs Caused by: java.nio.file.FileSystemException: C:\Workspace\kafka_2.11-1.1.0\kafka-logs\discrete-topic-0\00000000000000000000.log -> C:\Workspace\kafka_2.11-1.1.0\kafka-logs\discrete-topic-0\00000000000000000000.log.deleted: The process cannot access the file because it is being used by another process. [2018-08-01 18:14:01,504] INFO Stopping serving logs in dir C:\Workspace\kafka_2.11-1.1.0\kafka-logs (kafka.log.LogManager) [2018-08-01 18:14:01,508] ERROR Shutdown broker because all log dirs in C:\Workspace\kafka_2.11-1.1.0\kafka-logs have failed (kafka.log.LogManager) 

This seems to be this issue: https://issues.apache.org/jira/browse/KAFKA-6188

kafka version: kafka_2.11-1.1.0 Windows 10 Zookeeper version: 3.4.12

I have a few questions:

  1. Has anyone seen this issue and is there a workaround?
  2. Is running a kafka broker on Windows a viable options at this point? I know it is not supported for Windows and there seem to be multiple critical issues.

Thanks

like image 811
user3029642 Avatar asked Aug 02 '18 01:08

user3029642


People also ask

What happens when a Kafka broker fails?

If any broker fails, data should not be lost. For fault-tolerance purposes, the partition is replicated and stored in different brokers. If leader brokers fail, then the controller will elects one of the replicas as the leader.

What is Kafka log Dirs?

Kafka Topic is a logical grouping of one or more Kafka partitions. Each kafka partition is essentially (log) file/s on the disk. So the data you published kafka are stored in these files (logs) only. log. dirs tells kafka where to create these files.

What are all broker responsibilities in Kafka?

A Kafka broker receives messages from producers and stores them on disk keyed by unique offset. A Kafka broker allows consumers to fetch messages by topic, partition and offset. Kafka brokers can create a Kafka cluster by sharing information between each other directly or indirectly using Zookeeper.


2 Answers

step 1 : point to a new log directory in server.properties file and save the file

log.dirs=C:\Tools\kafka_2.11-2.1.0\kafka-test-logs

step 2 : start the kafka server again

/bin/windows/kafka-server-start.bat /config/server.properties

like image 71
Hari Chukkala Avatar answered Sep 20 '22 23:09

Hari Chukkala


I had the same kafka behavior but simply renaming or cleaning log directories wouldn't do the trick. And every time I tried to start kakfa it would crash (it was generating back all the logs deleted).

Solution:

  1. Delete the logs as before.
  2. Go to ZooKeeper data dir (you find it in \conf\zoo.cfg) and clean it. Restart everything and it should be completely reset.
like image 39
Tudor Avatar answered Sep 19 '22 23:09

Tudor