Want to understand the output of tablehistograms

Question

I wanted to understand the output of tablehistograms for a particular table and how can i calculate the cluster read/write, cluster latency using this for the whole cluster or Is there some other way to calculate cluster level read/write latency??

Here is one output of tablehistograms command :

bin/nodetool tablehistograms ks myTable;

Percentile  SSTables     Write Latency      Read Latency    Partition Size        Cell Count
                              (micros)          (micros)           (bytes)                  
50%             0.00             39.50             36.00              1597                42
75%             0.00             49.00             55.00              1597                42
95%             0.00             95.00             82.00              8239               258
98%             0.00            126.84            110.42             17084               446
99%             0.00            155.13            123.71             24601               770
Min             0.00              3.00              3.00              1110                36
Max             0.00          50772.00            314.00            126934              3973

Answer 1

The table histograms will give you the latencies for local reads/writes. When a request comes in the coordinator will distribute the read requests or mutations to the replicas, these metrics are just the time the replicas are spending applying the mutation or retrieving data and returning to coordinator.

nodetool proxyhistograms may be closer to what you are looking for, where thats the time the coordinator spends from time the request is received (more or less) til responding (more or less).

For more details you can query JMX (or other reporter you have configured) and get the org.apache.cassandra.metrics:type=Table,keyspace=<Keyspace>,scope=<Table>,name=CoordinatorReadLatency mbean to see how long reads have taken at coordinator level to that specific table.

What this means from our output:

• For writes to your table, 99% of mutations were applied on the replicas within 155 microseconds.
• Reading the data from disk and returning from coordinator finishes within 123 us 99% of the time, with a median of 36 us.
• 99% of the partitions are less than or equal to 24k, ~1.5k median
• 99% of partitions have 770 cells or less, 42 being the median (each field in a row is a cell)