Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Pandas: Filter dataframe by difference between adjacent rows

I have the following data in a dataframe.

Timestamp MeasureA MeasureB MeasureC MeasureD
0.00 26.46 63.60 3.90 0.67
0.94 26.52 78.87 1.58 0.42
1.94 30.01 82.04 1.13 0.46
3.00 30.19 82.00 1.17 0.36
4.00 30.07 81.43 1.13 0.42
5.94 30.02 82.46 1.05 0.34
8.00 30.22 82.48 0.98 0.35
9.00 30.00 82.21 1.13 0.33
10.00 30.00 82.34 1.12 0.34

And I'd like to filter the entries using some non-uniform intervals. Let say that my intervals are [1.0, 1.5]

What I'm trying to achieve is that, we take the first row (row0), and to get the next valid row, we look at what is the next row whose Timestamp value is greater or equal to row0 + 1.0.

In this scenario, the next valid row will be the one with the 1.94 timestamp. Then, for the next valid row, we will use the next item in the intervals array. Which is 1.5. That will make the next row the one with a timestamp value of 4.00. Since 1.94 + 1.5 is equal to 3.44.

For the next row, we go back and start from the beginning of the array of intervals.

After going through all the data, the resulting dataframe should be:

Timestamp MeasureA MeasureB MeasureC MeasureD
0.00 26.46 63.60 3.90 0.67
1.94 30.01 82.04 1.13 0.46
4.00 30.07 81.43 1.13 0.42
5.94 30.02 82.33 1.11 0.35
8.00 30.22 82.48 0.98 0.35
9.00 30.00 82.21 1.13 0.33

Is there a way to achieve this with the existing filtering methods in pandas?

like image 671
Jacobo Avatar asked Nov 19 '25 15:11

Jacobo


1 Answers

Try:

from itertools import cycle

# the interval:
A, B = 1.0, 1.5

comparing, out, last_t = cycle([B, A]), [], float("-inf")
j = next(comparing)

for i, t in zip(df.index, df.Timestamp):
    if t >= last_t + j:
        out.append(i)
        last_t = t
        j = next(comparing)

print(df.loc[out])

Prints:

   Timestamp  MeasureA  MeasureB  MeasureC  MeasureD
0       0.00     26.46     63.60      3.90      0.67
2       1.94     30.01     82.04      1.13      0.46
4       4.00     30.07     81.43      1.13      0.42
5       5.94     30.02     82.46      1.05      0.34
6       8.00     30.22     82.48      0.98      0.35
7       9.00     30.00     82.21      1.13      0.33
like image 178
Andrej Kesely Avatar answered Nov 21 '25 04:11

Andrej Kesely



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!