Cassandra server throws java.lang.AssertionError: DecoratedKey(...) != DecoratedKey

Question

I'm currently experimenting around with Cassandra.

On the client-side (with Hector) I look up a few keys like this:

ColumnFamilyResult<String, String> result = template.queryColumns(Arrays.asList("key1","key2","key3"));

Most of the time it seems to work. But other times I get a timeout exception on the client:

Caused by: me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
    at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
    at me.prettyprint.cassandra.service.template.ThriftColumnFamilyTemplate$1.execute(ThriftColumnFamilyTemplate.java:100)
    at me.prettyprint.cassandra.service.template.ThriftColumnFamilyTemplate$1.execute(ThriftColumnFamilyTemplate.java:88)
    at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
    at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
    at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
    at me.prettyprint.cassandra.service.template.ThriftColumnFamilyTemplate.sliceInternal(ThriftColumnFamilyTemplate.java:88)
    at me.prettyprint.cassandra.service.template.ThriftColumnFamilyTemplate.doExecuteSlice(ThriftColumnFamilyTemplate.java:46)
    at me.prettyprint.cassandra.service.template.ColumnFamilyTemplate.queryColumns(ColumnFamilyTemplate.java:113)
    at info.gamlor.experiments.Cassandra.readObjectByKey(ComplexCassandra.java:255)

Caused by: TimedOutException()
    at org.apache.cassandra.thrift.Cassandra$get_slice_result.read(Cassandra.java:7772)
    at org.apache.cassandra.thrift.Cassandra$Client.recv_get_slice(Cassandra.java:570)
    at org.apache.cassandra.thrift.Cassandra$Client.get_slice(Cassandra.java:542)
    at me.prettyprint.cassandra.service.template.ThriftColumnFamilyTemplate$1.execute(ThriftColumnFamilyTemplate.java:95)

And on the server this exception shows up:

ERROR 11:33:55,312 Exception in thread Thread[ReadStage:91,5,main]
java.lang.AssertionError: DecoratedKey(4948402862350542345439897754126541659, 6932) != DecoratedKey(132475956107784875457507977471906551877, 726f6f74) in C:\tem
p\cassandra\lib\cassandra\data\CassandraPolepos\ComplexObjects\CassandraPolepos-ComplexObjects-hd-2-Data.db
        at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:58)
        at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
        at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:78)
        at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:63)
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1331)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1193)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1128)
        at org.apache.cassandra.db.Table.getRow(Table.java:378)
        at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
        at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:816)
        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1250)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

Sometimes the key-values in the DecoratedKey(...) part takes up pages.

Anyone a hint what I'm doing wrong. Or how to investigate this issue.

Thanks.

Answer 1

This is either https://issues.apache.org/jira/browse/CASSANDRA-4687 or https://issues.apache.org/jira/browse/CASSANDRA-5202 You can run the command "nodetool invalidatekeycache" on all of the server showing the error. If it keeps coming back you may want to disable key cache. If it is 5202 it should only happen after dropping and recreating a column family using the same name.