Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to remove consecutive duplicate rows in KDB?

Tags:

kdb

q-lang

For instance, if I have the below table, then I want to remove the 3rd row:

Stock   Price
-------------------
GOOG    101
GOOG    102
GOOG    102     <- want to remove this
GOOG    101

Note: even though row 4 is a duplicate of row 1, I don't want to remove it as it is not a consecutive duplicate. That is, it is not a duplicate of the row immediately above.

I would also want to do check for duplicates across multiple fields, not just Price.

like image 376
mchen Avatar asked Dec 19 '22 16:12

mchen


2 Answers

d:([]Stock:4#`GOOG;Price:101 102 102 101)
q)d
Stock Price
-----------
GOOG  101
GOOG  102
GOOG  102
GOOG  101

q)d where not d~'prev d
Stock Price
-----------
GOOG  101
GOOG  102
GOOG  101
like image 174
jgleeson Avatar answered Feb 11 '23 07:02

jgleeson


You can also use differ

q)t:([]stock:4#`GOOG; price:101 102 102 101)
q)differ t
1101b
q)t where differ t
stock price
-----------
GOOG  101
GOOG  102
GOOG  101

now let's suppose there is a time column, as you indicate in your comment above

q)t:update time:til count i from t
q)t
stock price time
----------------
GOOG  101   0
GOOG  102   1
GOOG  102   2
GOOG  101   3
q)t where differ `stock`price#t
stock price time
----------------
GOOG  101   0
GOOG  102   1
GOOG  101   3

Now going back to the t without a time column, for simplicity. This gives you speed up over the alternative method proposed by @jgleeson (which I think is great, but a speed up is always welcomed so thought I'd share this regardless)

q)\ts do[10000;r:t where differ t]
31 1184j
q)\ts do[10000;r2:t where not t~'prev t]
62 1488j
q)r~r2
1b
like image 21
JPC Avatar answered Feb 11 '23 06:02

JPC