Why do my Cassandra hints files keep getting corrupted

Question

We're running multiple Kubernetes clusters, which run Cassandra. Our usual procedure when doing a rolling restart of the Cassandra pods is to log into each and submit a nodetool drain and then trigger a recreation of that pod. But often when the pods restart we get errors like

ERROR [HintsDispatcher:2] 2017-08-07 11:09:32,489 HintsDispatchExecutor.java:243 - Failed to dispatch hints file 5fdd139d-4465-4825-85ef-f380bddcb67d-1502100535128-1.hints: file is corrupted ({})

Those corrupt files prevent Cassandra from starting. Is there a way to tell Cassandra to flush all buffers and stop writing, before stopping it, to ensure there are no corrupt files left behind?

Answer 1

You can try to disable hinted handoff, or try to truncate hints after the drain:

nodetool truncatehints

If you care about consistency, run repair after the process.

Warning: If you are working with ANY consistency setting or RF=1, this may lead to some data loss.