Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Elegant way to reduce/filter redundant sample data with bitcode

Tags:

python

pandas

csv

I want to ask for ideas of an elegant(time saving) way to reduce redundant digitally sampled data. I got a .csv files with a structure like that:

time, data1, data2, data3
0.1, 1, 1, 0
0.2, 1, 1, 0
0.3, 1, 1, 0
0.4, 1, 0, 0
0.5, 1, 0, 0
0.6, 1, 0, 0
0.7, 0, 1, 0
0.8, 0, 1, 0

I want to reduce the incoming data in such manner that all unnecessary time steps are removed. E.g.

time, data1, data2, data3
0.1, 1, 1, 0 
0.4, 1, 0, 0
0.7, 0, 1, 0

The remaining data should just consist of the timesteps whenever the bitcode has changed. I already implemented a method importing the csv file with pandas and using the corresponding mask and compress functions. But the way I implemented it is very time consuming.

like image 961
Baum_jammer Avatar asked May 02 '26 18:05

Baum_jammer


1 Answers

If first column is index use:

df = df[df.ne(df.shift()).any(axis=1)]
print (df)
      data1  data2  data3
time                     
0.1       1      1      0
0.4       1      0      0
0.7       0      1      0

If first column is not index:

df1 = df.iloc[:, 1:]
df = df[df1.ne(df1.shift()).any(axis=1)]
print (df)
   time  data1  data2  data3
0   0.1      1      1      0
3   0.4      1      0      0
6   0.7      0      1      0
like image 175
jezrael Avatar answered May 04 '26 07:05

jezrael