Is there any way to have Cassandra safely remove tombstones *before* gc_grace_seconds have elapsed?

Question

I know early removal of tombstones is dangerous because it can cause deleted data to be resurrected, but if all replicas have confirmed deletion then such removal should be safe. For example, if a table has replication factor 3 and all 3 nodes containing the key have confirmed that they have the appropriate tombstone, it should be safe to perform a compaction in which the tombstones are removed because there would be no lingering copies of the data.

Is such safe removal of tombstones possible in Cassandra?

I would rather set gc_grace_seconds to infinity and rely on this type of safe compaction of tombstones than worry about the timing of nodetool repair and gc_grace_seconds.

Answer 1

No, it's not possible to remove tombstones without changing your gc_grace_seconds.

There are operations where all 3 replicas may ack the tombstone, remove it, then need it later on. Consider the case where you need to stream an earlier SSTable back into a cluster.

This type of manual tombstone removal will be significantly worse performance wise since you'll only be doing it periodically rather than constantly. You'll be reading extra data unnecessarily as well as constantly compacting tombstones that should be removed.

My recommendation is to set your gc_grace_seconds to something reasonable (10 days is fine) and schedule repairs using opscenter or cassandra-reaper.