Python Pandas rolling aggregate a column of lists

Question

I have a simple dataframe df with a column of lists lists. I would like to generate an additional column based on lists.

The df looks like:

import pandas as pd
lists={1:[[1]],2:[[1,2,3]],3:[[2,9,7,9]],4:[[2,7,3,5]]}
#create test dataframe
df=pd.DataFrame.from_dict(lists,orient='index')
df=df.rename(columns={0:'lists'})
df

          lists
1           [1]
2     [1, 2, 3]
3  [2, 9, 7, 9]
4  [2, 7, 3, 5]

I would like df to look like this:

df
Out[9]: 
          lists                 rolllists
1           [1]                       [1]
2     [1, 2, 3]              [1, 1, 2, 3]
3  [2, 9, 7, 9]     [1, 2, 3, 2, 9, 7, 9]
4  [2, 7, 3, 5]  [2, 9, 7, 9, 2, 7, 3, 5]

Basically I want to 'sum'/append the rolling 2 lists. Note that row 1, because I only have 1 list 1, rolllists is that list. But in row 2, I have 2 lists that I want appended. Then for row three, append df[2].lists and df[3].lists etc. I have worked on similar things before, reference this:Pandas Dataframe, Column of lists, Create column of sets of cumulative lists, and record by record differences.
In addition, if we can get this part above, then I want to do this in a groupby (so the example below would be 1 group for example, so for instance the df might look like this in the groupby):

  Group         lists                 rolllists
1     A           [1]                       [1]
2     A     [1, 2, 3]              [1, 1, 2, 3]
3     A  [2, 9, 7, 9]     [1, 2, 3, 2, 9, 7, 9]
4     A  [2, 7, 3, 5]  [2, 9, 7, 9, 2, 7, 3, 5]
5     B           [1]                       [1]
6     B     [1, 2, 3]              [1, 1, 2, 3]
7     B  [2, 9, 7, 9]     [1, 2, 3, 2, 9, 7, 9]
8     B  [2, 7, 3, 5]  [2, 9, 7, 9, 2, 7, 3, 5]

I have tried various things like df.lists.rolling(2).sum() and I get this error:

TypeError: cannot handle this type -> object 

in Pandas 0.24.1 and unfortunatley in Pandas 0.22.0 the command doesn't error, but instead returns the exact same values as in lists. So Looks like newer versions of Pandas can't sum lists? That's a secondary issue.

Love any help! Have Fun!

Answer 1

You can start with

import pandas as pd
mylists={1:[[1]],2:[[1,2,3]],3:[[2,9,7,9]],4:[[2,7,3,5]]}
mydf=pd.DataFrame.from_dict(mylists,orient='index')
mydf=mydf.rename(columns={0:'lists'})
mydf = pd.concat([mydf, mydf], axis=0, ignore_index=True)
mydf['group'] = ['A']*4 + ['B']*4

# initialize your new series
mydf['newseries'] = mydf['lists']

# define the function that appends lists overs rows
def append_row_lists(data):
    for i in data.index:
        try: data.loc[i+1, 'newseries'] = data.loc[i, 'lists'] + data.loc[i+1, 'lists']
        except: pass
    return data

# loop over your groups
for gp in mydf.group.unique():
    condition = mydf.group == gp
    mydf[condition] = append_row_lists(mydf[condition])

Output

          lists Group                 newseries
0           [1]     A                       [1]
1     [1, 2, 3]     A              [1, 1, 2, 3]
2  [2, 9, 7, 9]     A     [1, 2, 3, 2, 9, 7, 9]
3  [2, 7, 3, 5]     A  [2, 9, 7, 9, 2, 7, 3, 5]
4           [1]     B                       [1]
5     [1, 2, 3]     B              [1, 1, 2, 3]
6  [2, 9, 7, 9]     B     [1, 2, 3, 2, 9, 7, 9]
7  [2, 7, 3, 5]     B  [2, 9, 7, 9, 2, 7, 3, 5]