How to apply by chunks a function to pandas dataframe?

Question

I know that apply() function can be use to apply a function to a dataframe's column:

df.applymap(my_fun)

How can I apply my_fun by chunks?, for instance chunks of 1, 5, 10, and 20 rows?.

Answer 1

IIUC you can use np.split() method:

In [5]: df
Out[5]:
     a   b   c
0   66  64  76
1   74  15   8
2    1  29  45
3   88  98  55
4   73  24  39
5   43  86  54
6    3  10  99
7    7  36  87
8    2  26   2
9   26  49  82
10  16  38  79
11  93  78  23
12  47  35  70
13  10  47  65
14  98  81  67
15  81  13  14
16  25  47  33
17  94  48  66
18  80  52  90
19  43  40  32

In [6]: df.shape
Out[6]: (20, 3)

In [7]: [x.shape for x in np.split(df, [5, 10, 18])]
Out[7]: [(5, 3), (5, 3), (8, 3), (2, 3)]

or if you need evenly sized chunks:

In [10]: [x.shape for x in np.split(df, np.arange(5, len(df), 5))]
Out[10]: [(5, 3), (5, 3), (5, 3), (5, 3)]

Using apply() function:

In [15]: [x.apply(np.sum) for x in np.split(df, np.arange(5, len(df), 5))]
Out[15]:
[a    302
 b    230
 c    223
 dtype: int64, a     81
 b    207
 c    324
 dtype: int64, a    264
 b    279
 c    304
 dtype: int64, a    323
 b    200
 c    235
 dtype: int64]

How does pandas apply a function?

Pandas applies given function to the given vector of values:

• it applies given function to the whole column vector (column by column) if axis=0
• it applies given function to the row vector (row by row) if axis=1