How to recover from Apache Cassandra CommitLogReplayException

Question

I had an unexpected shutdown of Mac OS X while running apache-cassandra 3.3. When restarting cassandra in a new bootload, I got the following error:

INFO  15:25:15 Replaying /Users/eswenson/opt/apache-cassandra-3.5/bin/../data/commitlog/CommitLog-6-1461717351786.log, /Users/eswenson/opt/apache-cassandra-3.5/bin/../data/commitlog/CommitLog-6-1461717351787.log, /Users/eswenson/opt/apache-cassandra-3.5/bin/../data/commitlog/CommitLog-6-1461942807785.log, /Users/eswenson/opt/apache-cassandra-3.5/bin/../data/commitlog/CommitLog-6-1461943004975.log
ERROR 15:25:15 Exiting due to error while processing commit log during initialization.
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Encountered bad header at position 542295 of commit log /Users/eswenson/opt/apache-cassandra-3.5/bin/../data/commitlog/CommitLog-6-1461717351786.log, with bad position but valid CRC
        at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:611) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.db.commitlog.SegmentReader$SegmentIterator.computeNext(SegmentReader.java:105) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.db.commitlog.SegmentReader$SegmentIterator.computeNext(SegmentReader.java:78) [apache-cassandra-3.5.jar:3.5]
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) [guava-18.0.jar:na]
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) [guava-18.0.jar:na]
        at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:399) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:236) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:192) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:172) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:283) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680) [apache-cassandra-3.5.jar:3.5]

I searched on stack overflow and google and did not find posts of this same problem. I found issue https://issues.apache.org/jira/browse/CASSANDRA-9749, which is related, but this one is marked resolved and is related to continuing execution after an problem is detected rather than failing with an exception.

What do I do in this situation? Is there any recovery path (assuming no snapshots avaialble)? Any way to truncate the commit log at the offending record? Any tool available to fix the bad header (e.g. delete the record)?

Thanks.

Answer 1

I also faced this exception sometimes when playing with my local Cassandra on Mac OS.

If you don't care about loosing the data in the commit logs or if you have a backup somewhere, just stop Cassandra, clean the commitlog folder and restart.

I'm afraid that there is no other alternative. By the way, are you deploying Cassandra on Mac OS for production ?

• If yes, you should know that Mac OS is not officially supported
• If it's not for production then who care, just clean the commitlog and restart

Answer 2

I faced similar kind of problem while running Cassandra on Windows 7 (local machine) environment. The exception I found from log is

ERROR 07:14:35 Exiting due to error while processing commit log during initialization. org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Could not read commit log descriptor in file C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1470969948498.log at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:616) [apache-cassandra-3.7.0.jar:3.7.0] at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:378) [apache-cassandra-3.7.0.jar:3.7.0]

As it was just a test setup I deleted that particular commit log file and restarted it from windows service manager and it started perfectly. I'm not sure what to do if it happens on production environment.