Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Sum a list of Pandas DataFrames

Is there a way to sum multiple pandas DataFrames using syntax similar to pd.concat([df1, df2, df3, df4]). I understand from documentation that I can do df1.sum(df2, fill_value=0), but I have a long list of DataFrames I need to sum and was wondering if I could do it without writing a loop.

Somewhat related question/answer: Pandas sum multiple dataframes (Stack Overflow)

Example of what the result should look like:

idx1 = pd.MultiIndex.from_tuples([('a', 'A'), ('a', 'B'), ('b', 'A'), ('b', 'D')])
idx2 = pd.MultiIndex.from_tuples([('a', 'A'), ('a', 'C'), ('b', 'A'), ('b', 'C')])
idx3 = pd.MultiIndex.from_tuples([('a', 'A'), ('a', 'D'), ('b', 'A'), ('b', 'C')])

np.random.seed([3,1415])
df1 = pd.DataFrame(np.random.randn(4, 1), idx1, ['val'])
df2 = pd.DataFrame(np.random.randn(4, 1), idx2, ['val'])
df3 = pd.DataFrame(np.random.randn(4, 1), idx3, ['val'])

df1

enter image description here

df2

enter image description here

df3

enter image description here

The result should look like:

enter image description here

like image 451
blahblahblah Avatar asked Aug 31 '17 13:08

blahblahblah


People also ask

How do I sum multiple Dataframes in pandas?

Pandas DataFrame sum() MethodThe sum() method adds all values in each column and returns the sum for each column. By specifying the column axis ( axis='columns' ), the sum() method searches column-wise and returns the sum of each row.

How do I sum a list of columns in pandas?

To sum given or list of columns then create a list with all columns you wanted and slice the DataFrame with the selected list of columns and use the sum() function. Use df['Sum']=df[col_list]. sum(axis=1) to get the total sum.

How do I sum across rows in pandas?

To sum all the rows of a DataFrame, use the sum() function and set the axis value as 1. The value axis 1 will add the row values.


1 Answers

Use reduce with add with parameter fill_value=0:

np.random.seed(12)

a = pd.DataFrame(np.random.randint(3, size=(5,3)), columns=list('abc'))
b = pd.DataFrame(np.random.randint(3, size=(5,2)), columns=list('ab'))
c = pd.DataFrame(np.random.randint(3, size=(5,2)), columns=list('ac'))
print(a)
   a  b  c
0  2  1  1
1  2  0  0
2  2  1  0
3  1  1  1
4  2  2  2

print(b)
   a  b
0  0  1
1  0  0
2  1  2
3  1  2
4  0  1

print(c)
   a  c
0  2  0
1  2  2
2  2  0
3  0  2
4  1  1

from functools import reduce

dfs = [a,b, c]
d = reduce(lambda x, y: x.add(y, fill_value=0), dfs)
print (d)
   a    b    c
0  4  2.0  1.0
1  4  0.0  2.0
2  5  3.0  0.0
3  2  3.0  3.0
4  3  3.0  3.0
like image 137
jezrael Avatar answered Oct 16 '22 18:10

jezrael