OPENTSDB fsck --fix not correcting duplicate points

Question

I am using OPENTSDB and while querying I am getting this:

net.opentsdb.core.IllegalDataException: Found out of order or duplicate data: cell=Cell([-35, 87], [0, 0, 0, 0, 0, 8, -34, 65]), delta=3541, prev cell=Cell([-35, 87], [0, 0, 0, 0, 0, 12, -82, 106]), last_delta=3541, in row=[KeyValue(key=[0, 8, -96, 81, -7, -77, 16, 0, 0, 1, 0, -73, 83, 0, 0, 3, 0, 47, 57, 0, 0, 69, 0, 44, 99, 0, 0, 71, 0, 48, 79, 0, 0, 75, 0, 47, -53, 0, 0, 76, 0, 13, -24, 0, 0, 77, 0, 114, 14, 0, 0, 85, 0, -16, -50], family="t", qualifier="\xDDW", value=[0, 0, 0, 0, 0, 12, -82, 106], timestamp=1375323607530), KeyValue(key=[0, 8, -96, 81, -7, -77, 16, 0, 0, 1, 0, -73, 83, 0, 0, 3, 0, 47, 57, 0, 0, 69, 0, 44, 99, 0, 0, 71, 0, 48, 79, 0, 0, 75, 0, 47, -53, 0, 0, 76, 0, 13, -24, 0, 0, 77, 0, 114, 14, 0, 0, 85, 0, -16, -50], family="t", qualifier=[-35, 87, -35, -41, -34, 103, -32, 7, -32, -57], value=[0, 0, 0, 0, 0, 8, -34, 65, 0, 0, 0, 0, 0, 1, -122, -123, 0, 0, 0, 0, 0, 3, -22, 23, 0, 0, 0, 0, 0, 13, -10, -32, 0, 0, 0, 0, 0, 10, -27, 6, 0], timestamp=1375323057833)] -- run an fsck.

I have tried using fsck --fix but that is saying no errors found. Is there a way to: 1. resolve this apart from removing the datpoints manually 2. understanding what happening and how to prevent this.

Thanks

Answer 1

Solution 1: to avoid this to happen from start, set the tsd.storage.fix_duplicates flag to true in your opentsdb.conf.

---

Solution 2: In case you already have duplicate values written to your Hbase (the underlying datastore), and are unable to query the openTSDB - use the fsck utility: while inside opentsdb/build/

Specific query:

 ./tsdb fsck --fix-all 1h-ago now sum <metric-name> tag1=val1

For metric:

./tsdb fsck --threads=2 --fix-all --resolve-duplicates 15d-ago sum <metric name>

Full Table: all the data in the Hbase's 'tsdb' table (the one table openTSDB stores data)

./tsdb fsck --full-scan --threads=8 --fix-all --resolve-duplicates --compact

The helpful fsck flags:

• --fix-all - Sets all repair flags to attempt to fix all issues at once. Use with caution.
• --compact Compacts non-compacted rows during a repair.
• --delete-bad-compacts Removes columns that appear to be compacted but failed parsing. If a column parses properly but the final byte of the value is not set to a 0 or a 1, the column will be left alone.
• --resolve-duplicates Enables duplicate data point resolution by deleting all but the latest or oldest data point. Also see --last-write-wins.
• --last-write-wins Flag When set, deletes all but the most recently written data point when resolving duplicates. If the config value tsd.storage.fix_duplicates is set to true, then the latest data point will be kept regardless of this value. Not Set --last-write-wins
• --full-scan Scans the entire data table. Note: This can take a very long time to complete.
• --threads Integer The number of threads to use when performing a full scan. The default is twice the number of CPU cores.