Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Forward Fill Pandas Dataframe Horizontally (along rows) without forward filling last value in each row

I have a Pandas dataframe that I want to forward fill HORIZONTALLY but I don't want to forward fill past the last entry in each row. This is time series pricing data on products where some have been discontinued so I dont want the last value recorded to be forward filled to current.

FWDFILL.apply(lambda series: series.iloc[:,series.last_valid_index()].ffill(axis=1))

^The code I have included does what I want but it does it VERTICALLY. This could maybe help people as a starting point.

>>> print(FWDFILL)

1      1     NaN     NaN     2     NaN  
2     NaN     1      NaN     5     NaN  
3     NaN     3       1     NaN    NaN  
4     NaN    NaN     NaN    NaN    NaN  
5     NaN     5      NaN    NaN     1  

Desired Output:

1      1      1      1     2     NaN  
2     NaN     1      1     5     NaN  
3     NaN     3      1    NaN    NaN  
4     NaN    NaN    NaN   NaN    NaN  
5     NaN     5      5     5      1
like image 561
Fojomojo Avatar asked Jun 13 '19 19:06

Fojomojo


2 Answers

IIUC, you need to apply with axis=1, so you are applying to dataframe rows instead of dataframe columns.

df.apply(lambda x: x[:x.last_valid_index()].ffill(), axis=1)

Output:

     1    2    3    4    5
0                         
1  1.0  1.0  1.0  2.0  NaN
2  NaN  1.0  1.0  5.0  NaN
3  NaN  3.0  1.0  NaN  NaN
4  NaN  NaN  NaN  NaN  NaN
5  NaN  5.0  5.0  5.0  1.0
like image 75
Scott Boston Avatar answered Oct 21 '22 21:10

Scott Boston


Usage of bfill and ffill

s1=df.ffill(1)
s2=df.bfill(1)
df=df.mask(s1.notnull()&s2.notnull(),s1)
df
Out[222]: 
     1    2    3    4    5
1  1.0  1.0  1.0  2.0  NaN
2  NaN  1.0  1.0  5.0  NaN
3  NaN  3.0  1.0  NaN  NaN
4  NaN  NaN  NaN  NaN  NaN
5  NaN  5.0  5.0  5.0  1.0

Or just using interpolate

df.mask(df.interpolate(axis=1,limit_area='inside').notnull(),df.ffill(1))
Out[226]: 
     1    2    3    4    5
1  1.0  1.0  1.0  2.0  NaN
2  NaN  1.0  1.0  5.0  NaN
3  NaN  3.0  1.0  NaN  NaN
4  NaN  NaN  NaN  NaN  NaN
5  NaN  5.0  5.0  5.0  1.0
like image 27
BENY Avatar answered Oct 21 '22 21:10

BENY