Forward fill all except last value in python pandas dataframe

Question

I have a dataframe in pandas with several columns I want to forward fill the values for. At the moment I'm doing:

columns = ['a', 'b', 'c']
for column in columns:
    df[column].fillna(method='ffill', inplace=True)

...but because the series in the columns are different lengths, that leaves long tails of filled values on the ends of some of them. Because the gaps in the some of the series are quite large, I can't use the fillna's limit parameter without also leaving long tails of filled values on the series.

Is it possible to forward fill the values in each columns, except the last value? Thanks!

Answer 1

You can use last_valid_index in a lambda function to just ffill up to that point.

df = pd.DataFrame({
    'A': [1, None, None, None], 
    'B': [1, 2, None, None], 
    'C': [1, None, 3, None], 
    'D': [1, None, None, 4]})

>>> df
    A   B   C   D
0   1   1   1   1
1 NaN   2 NaN NaN
2 NaN NaN   3 NaN
3 NaN NaN NaN   4

>>> df.apply(lambda series: series.loc[:series.last_valid_index()].ffill())
    A   B   C  D
0   1   1   1  1
1 NaN   2   1  1
2 NaN NaN   3  1
3 NaN NaN NaN  4

Answer 2

In addition to the answer from Alexander, you can use the following if you want to conserve bottom rows with NaNs:

df2 = pd.DataFrame({
    'A': [1, None, None, None, None], 
    'B': [1, 2, None, None, None], 
    'C': [1, None, 3, None, None], 
    'D': [1, None, None, 4, None]})

df2
    A   B   C   D
0   1   1   1   1
1 NaN   2 NaN NaN
2 NaN NaN   3 NaN
3 NaN NaN NaN   4
4 NaN NaN NaN NaN

pd.concat([df2.apply(lambda series: series.loc[:series.last_valid_index()].ffill()),
           df2.loc[df2.last_valid_index()+1:]])

    A   B   C   D
0   1.0 1.0 1.0 1.0
1   NaN 2.0 1.0 1.0
2   NaN NaN 3.0 1.0
3   NaN NaN NaN 4.0
4   NaN NaN NaN NaN