Pandas' expanding with apply function on multiple columns

Question

Is it possible to use panda's expanding function to calculate the coefficient of a polynomial regression using several columns of the window object?

I have a data frame which has two columns, a predictor and a response. I want to use pandas' expanding() function to calculate the corresponding coefficients of a second order polynomial regression for each expanding pair of series. For each row I would like to get the updated coefficients from the regression applied to all previous rows.

import pandas as pd
import numpy as np

def func1(df):
   # some processing
   return np.polyfit(df['Input'], df['Response'], 2)
   
def func2(x, y):
   # some processing
   return np.polyfit(x, y, 2)

np.random.seed(0)
df = pd.DataFrame(np.random.rand(10, 2).round(2), 
                  columns=['Input', 'Response'])

df[['Coef1', 'Coef2', 'Coef3']] = df.expanding(min_periods=3).apply(func)

I'd like the output to look like this:

>>> df

   Input Response Coef1  Coef2  Coef3
0  0.63  0.23     NaN    NaN    NaN
1  0.45  0.11     NaN    NaN    NaN
2  0.17  0.71     NaN    NaN    NaN
3  0.17  0.32     0.19   0.54   0.50
4  0.65  0.99     0.48   0.23   0.60
5  0.21  0.54     0.71   0.89   0.97
6  0.63  0.73     0.22   0.05   0.80
7  0.54  0.23     0.87   0.01   0.25
8  0.33  0.06     0.18   0.96   0.03
9  0.18  0.72     0.13   0.38   0.13

My different trials has led to two types of error. If I use the function that uses the dataframe as a parameter such as in df[['Coef1', 'Coef2', 'Coef3']] = df.expanding(min_periods=3).apply(func1)), I get KeyError: 'Input'. If I use the second function where I extract the parameters before df['Coef1', 'Coef2', 'Coef3'] = df.expanding(min_periods=3).apply(lambda x: func2(x['Input'], x['Output'])), I get DataError: No numeric types to aggregate However, If I try for instance df.expanding().cov(pairwise=True) it shows that calculation can be performed on the different columns of the object returned by expanding. There's a similar question here: Apply expanding function on dataframe. However, the solution consisting in calling expanding() in the function does not seem to apply in this case. I would appreciate any pointers or suggestion.

I found a package that does that with numpy so it inspired me to do it manually:

def func_np(df):
    length = len(df)
    if length == 1:
        return [[0], [0], [0]]

    coef1, coef2, coef3 = [], [], []

    x = df['A'].to_numpy()  # This is the predictor column
    y = df['B'].to_numpy()  # This is the response column

    for step in range(1, length + 1):
        weights = np.polyfit(x[: step], y[: step], 2)  # 2 is the polynomial's order
        coef1.append(weights[0])
        coef2.append(weights[1])
        coef3.append(weights[2])
    # Note that coef1, coef2, coef3 correspond to the polynomial terms from highest to lowest

    # It is easier to return a data frame, so that we can reassign the result to the initial one
    return pd.DataFrame({'Coef1': coef1, 'Coef2': coef2, 'Coef3': coef3})

I wanted to do it with Numba to speed up the execution but it does not recognize the np.polyfit function. Also I have not found a neat way to assign back the results to the initial data frame. That is why I am still interested in seeing a simple and more "pythonic" solution with expanding()

Answer 1

I suspect what you are looking for is the new df.expanding(..., method='table') in the upcoming pandas=1.3 (see "Other enhancements").

In the meantime, you can do it "by hand", using a loop (sorry):

xy = df.values
df['c1 c2 c3'.split()] = np.stack([
    func2(*xy[:n].T) if n >= 3 else np.empty(3)*np.nan
    for n in range(xy.shape[0])
])

Example:

np.random.seed(0)
df = pd.DataFrame(np.random.rand(10, 2).round(2), 
                  columns=['Input', 'Response'])

# the code above, then

>>> df
   Input  Response         c1         c2        c3
0   0.55      0.72        NaN        NaN       NaN
1   0.60      0.54        NaN        NaN       NaN
2   0.42      0.65        NaN        NaN       NaN
3   0.44      0.89 -22.991453  22.840171 -4.887179
4   0.96      0.38 -29.759096  29.213620 -6.298277
5   0.79      0.53   0.454036  -1.369701  1.272156
6   0.57      0.93   0.122450  -0.874260  1.113586
7   0.07      0.09  -1.010312   0.623331  0.696287
8   0.02      0.83  -2.687387   2.995143 -0.079214
9   0.78      0.87  -1.425030   1.294210  0.442684