Cassandra nodes randomly going down

Question

I am running into problems with Cassandra 1.2.5 nodes going into high CPU usage 2300% and random becoming unresponsive (for example cannot log into cqlsh, getting timout exception).

I am not sure why but while running tpstats on that node i have 26000 pending ReadStage pool and FlushWriter 126 all time blocked. It seems that after restarting the node things come back to normal for a while and after period of time the problem reappears and I get timeouts all over the place and the data is not retrieved correctly from the table that were not affected by the deployment.

I have deployed modification to the application prior to the problem identification where I introduced the table that receives heavy writes and after the data is read it is deleted. I wonder if the problem is due to compaction can anyone help???

Answer 1

If FlushWriter is (ever) blocking that is a very bad sign. Usually indicates that you're trying to do far more with your hardware than you have capacity for. If I were to make a wild guess: your CPU is being chewed up by GC and that's hosing everything.

26000 pending reads alone is probably enough to blow your heap. Hard to say from your description what the problem is, but I would suspect your IO subsystem is overloaded causing reads to back up (this is where your problem begins). Those read requests end up on the heap and sit there until the request times out. Your pending reads blow up your heap and you end up in a GC death spiral.

If this ^ is your problem: add capacity by using more nodes, or use SSDs.