Cassandra Exception Cache schema version X does not match current schema version Y

Question

We had DataStax Cassandra Community Server 3.0.2 installed on our boxes, got couple of keyspaces created and we migrated some data from our old DB to Cassandra (Aroudnd 10GB of data). While, that was migration was happening we created materialized views on the those tables. Some minutes( not sure exactly how long) after the creation, we got our services(deployed on ~2x3 nodes in 2 Data Center Architecture) randomly stopped with absolutely NO log information on why it stopped. After that we tried to start our service, we got the following error in our logs

java.lang.RuntimeException: Cache schema version a2c390a1-f2cd-3d59-8b5c-a0a1d794d0fe does not match current schema version 8d1347d7-7729-3698-8537-4b91ae9ce7dd
    at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:198) ~[apache-cassandra-3.0.2.jar:3.0.2]
    at org.apache.cassandra.cache.AutoSavingCache$3.call(AutoSavingCache.java:157) [apache-cassandra-3.0.2.jar:3.0.2]
    at org.apache.cassandra.cache.AutoSavingCache$3.call(AutoSavingCache.java:153) [apache-cassandra-3.0.2.jar:3.0.2]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_66]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_66]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_66]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66] 

We tried to look into the code for where this excpetion might be caused. This gave us some information on what causes it - apparently the service tries to match the current schema version with the one that present in cache(I would assume its either before the service stopped/creation of materialized views) We are very sure that we didnt do any schema changes on the keyspace apart from creating Materialized Views.

So what makes me wonder is no-one has report any sort of such issue in the past ? So we are not sure what goes wrong on our side. We ran nodetool cleanup on our nodes with no help. We don't want to lose any data. We are trying to recover, so we deleted the $CASSANDRA_HOME\data\saved_caches folder and restarting the service.While starting up the server it is replaying the commitlogs and it is taking long time. I assume that might be due to the amount of data we have or machines are totally gone and need to be replaced ?

Any pointers related to the issue might be really helpful! Thank you in advance.

If it might help, we are running DataStax Cassandra Server on Windows Machines and running Cassandra as a windows service.

Answer 1

Clearing the commitlog directory and restarting it did it for me.

rm -rf /path/to/cassandra/data/commitlog

Late but I hope it helps anyone who's stuck.

Answer 2

Cache schema version ...

Clear the saved_caches directory not the commit_log directory. Deleting commit_logs is dangerous. Commit logs can save the day and prevent data loss if your nodes crashed unexpectedly and your memtable data isn't flushed.