Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Pandas replace last item in groupby if NaN with another colum

I'm trying to replace the last row in a group by with the value of another column only if it is null. I am able to do both of these pieces separately but can't seem to combine them. Anyone have any ideas?

These are the separate pieces:

# replace any NaN values with values from 'target'
df.loc[df['target'].isnull(),'target'] = df['value']

# replace last value in groupby with value from 'target'
df.loc[df.groupby('id').tail(1).index,'target'] = df['value']

Original Data:

    date        id      value       target
0   2020-08-07  id01    0.100775    NaN
1   2020-08-08  id01    0.215885    0.215885
2   2020-08-09  id01    0.012154    0.012154
3   2020-08-10  id01    0.374503    NaN
4   2020-08-07  id02    0.369707    0.369707
5   2020-08-08  id02    0.676743    0.676743
6   2020-08-09  id02    0.659521    0.659521
7   2020-08-10  id02    0.799071    NaN

Replace 'target' column with last row in groupby('id') with what is in 'value':

    date        id      value       target
0   2020-08-07  id01    0.100775    NaN
1   2020-08-08  id01    0.215885    0.215885
2   2020-08-09  id01    0.012154    0.012154
3   2020-08-10  id01    0.374503    0.374503
4   2020-08-07  id02    0.369707    0.369707
5   2020-08-08  id02    0.676743    0.676743
6   2020-08-09  id02    0.659521    0.659521
7   2020-08-10  id02    0.799071    0.799071
like image 582
dingo Avatar asked Jan 24 '23 20:01

dingo


1 Answers

This should do. Added the tail variable just for easier to read syntaxis:

tail = df.groupby('id').tail(1)
df.loc[tail.index,'target'] = df.loc[tail.index]['target'].fillna(tail.value) 

Output:

0 idx        date    id     value    target
1   0  2020-08-07  id01  0.100775       NaN
2   1  2020-08-08  id01  0.215885  0.215885
3   2  2020-08-09  id01  0.012154  0.012154
4   3  2020-08-10  id01  0.374503  0.374503
5   4  2020-08-07  id02  0.369707  0.369707
6   5  2020-08-08  id02  0.676743  0.676743
7   6  2020-08-09  id02  0.659521  0.659521
8   7  2020-08-10  id02  0.799071  0.799071
like image 146
Juan C Avatar answered Jan 27 '23 10:01

Juan C