Rolling a function on a data frame

Question

I have the following data frame C.

>>> C
              a    b   c
2011-01-01    0    0 NaN
2011-01-02   41   12 NaN
2011-01-03   82   24 NaN
2011-01-04  123   36 NaN
2011-01-05  164   48 NaN
2011-01-06  205   60   2
2011-01-07  246   72   4
2011-01-08  287   84   6
2011-01-09  328   96   8
2011-01-10  369  108  10

I would like to add a new column, d, where I apply a rolling function, on a fixed window (6 here), where I somehow, for each row (or date), fix the value c. One loop in this rolling function should be (pseudo):

              a    b   c   d
2011-01-01    0    0 NaN   a + b*2 (a,b from this row, '2' is from 'c' on 2011-01-06)
2011-01-02   41   12 NaN   a + b*2 (a,b from this row, '2' is still from 2011-01-06)
2011-01-03   82   24 NaN   a + b*2
2011-01-04  123   36 NaN   a + b*2
2011-01-05  164   48 NaN   a + b*2
2011-01-06  205   60   2   a + b*2
2011-01-07  246   72   4   
2011-01-08  287   84   6   
2011-01-09  328   96   8   
2011-01-10  369  108  10

After this "loop" I want to take all of these 6 calculated rows in d and run a function call, which in turn will return one value, that should be stored in another column, e say:

              a    b   c   d                               e

2011-01-01    0    0 NaN   a + b*2 ---|                   NaN
2011-01-02   41   12 NaN   a + b*2    |                   NaN
2011-01-03   82   24 NaN   a + b*2    | These values      NaN
2011-01-04  123   36 NaN   a + b*2    | are input to      NaN
2011-01-05  164   48 NaN   a + b*2    | function          NaN
2011-01-06  205   60   2   a + b*2 ---| yielding          X
2011-01-07  246   72   4                value X in
2011-01-08  287   84   6                column 'e'
2011-01-09  328   96   8   
2011-01-10  369  108  10

This procedure would then be iterated onto the next window (again 6 long) like:

              a    b   c   d             e
2011-01-01    0    0 NaN   
2011-01-02   41   12 NaN   a + b*4 (a,b from this row, '4' is from 'c' now from 2011-01-07)
2011-01-03   82   24 NaN   a + b*4 (a,b from this row, '4' is still from 2011-01-07)
2011-01-04  123   36 NaN   a + b*4
2011-01-05  164   48 NaN   a + b*4
2011-01-06  205   60   2   a + b*4       X
2011-01-07  246   72   4   a + b*4
2011-01-08  287   84   6   
2011-01-09  328   96   8   
2011-01-10  369  108  10

              a    b   c   d                               e

2011-01-01    0    0 NaN                                  NaN
2011-01-02   41   12 NaN   a + b*4 ---|                   NaN
2011-01-03   82   24 NaN   a + b*4    | These values      NaN
2011-01-04  123   36 NaN   a + b*4    | are input to      NaN
2011-01-05  164   48 NaN   a + b*4    | function          NaN
2011-01-06  205   60   2   a + b*4    | yielding          X
2011-01-07  246   72   4   a + b*4 ---| value Y in        Y
2011-01-08  287   84   6                column 'e'
2011-01-09  328   96   8   
2011-01-10  369  108  10

Hopefully this is clear enough,

Thanks, N

Answer 1

You could use pd.rolling_apply:

import numpy as np
import pandas as pd
df = pd.read_table('data', sep='\s+')

def foo(x, df):
    window = df.iloc[x]
    # print(window)
    c = df.ix[int(x[-1]), 'c']
    dvals = window['a'] + window['b']*c
    return bar(dvals)

def bar(dvals):
    # print(dvals)
    return dvals.mean()

df['e'] = pd.rolling_apply(np.arange(len(df)), 6, foo, args=(df,))
print(df)

yields

              a    b   c       e
2011-01-01    0    0 NaN     NaN
2011-01-02   41   12 NaN     NaN
2011-01-03   82   24 NaN     NaN
2011-01-04  123   36 NaN     NaN
2011-01-05  164   48 NaN     NaN
2011-01-06  205   60   2   162.5
2011-01-07  246   72   4   311.5
2011-01-08  287   84   6   508.5
2011-01-09  328   96   8   753.5
2011-01-10  369  108  10  1046.5

---

The args and kwargs parameters were added to rolling_apply in Pandas version 0.14.0.

Since in my example above df is a global variable, it is not really necessary to pass it to foo as an argument. You could simply remove df from the def foo line and also omit the args=(df,) in the call to rolling_apply.

However, there are times when df might not be defined in a scope accessible by foo. In that case, there is a simple workaround -- make a closure:

def foo(df):
    def inner_foo(x):
        window = df.iloc[x]
        # print(window)
        c = df.ix[int(x[-1]), 'c']
        dvals = window['a'] + window['b']*c
        return bar(dvals)
    return inner_foo

df['e'] = pd.rolling_apply(np.arange(len(df)), 6, foo(df))