decipher dropped mutations message

Question

How do I read dropped mutations error messages - what is internal and cross node? For mutations it fails on cross-node and read_repair/read it fails on internal. What does it mean?

INFO  [ScheduledTasks:1] 2019-07-21 11:44:46,150  MessagingService.java:1281 - MUTATION messages were dropped in last 5000 ms: 0 internal and 65 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 4966 ms
INFO  [ScheduledTasks:1] 2019-07-19 05:01:10,620  MessagingService.java:1281 - READ_REPAIR messages were dropped in last 5000 ms: 9 internal and 8 cross node. Mean internal dropped latency: 6013 ms and Mean cross-node dropped latency: 8164 ms

Does internal mean local node operations?
In this case, does mutation response from local node and cross node means the time it took to get response back from other nodes depending on the consistency level chosen?

What does Read and _Trace dropped mutations mean? There is no tracing enabled on any node in the cluster. What are these _TRACE dropped messages?

INFO  [ScheduledTasks:1] 2019-07-25 21:17:13,878  MessagingService.java:1281 - READ messages were dropped in last 5000 ms: 1 internal and 0 cross node. Mean internal dropped latency: 5960 ms and Mean cross-node dropped latency: 0 ms
INFO  [ScheduledTasks:1] 2019-07-25 20:38:43,788  MessagingService.java:1281 - _TRACE messages were dropped in last 5000 ms: 5035 internal and 0 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 0 ms

Answer 1

How do I read dropped mutations error messages

You can use JMX to read new dropped messages but you will not be able to check the messages which are already dropped. You can enable JMX using this link.

In JMX these are accessible via org.apache.cassandra.net:MessagingService or org.apache.cassandra.metrics:DroppedMessage.

What does Read and _Trace dropped mutations mean?

Read messages are messages corresponding to your actual read request.Read request times out after read_request_timeout_in_ms. No point in servicing reads after that point since it would of returned error to the client.

Trace is used for recording traces (nodetool settraceprobability). It has a special executor (1 thread, 1000 queue depth) that throws away messages on insertion instead of within the execute

Check your traceprobability by nodetool gettraceprobability.

Does internal mean local node operations?

Yes, you are right. Internal node means local node operations.

If you are seeing a lot of MUTATION failure, it means a lot of writes are failing due to timeouts and you may need to check your Cassandra servers and increase infra if necessary.