Pandas concat failing

Question

I am trying to concat dataframes based on the foll. 2 csv files:

df_a: https://www.dropbox.com/s/slcu7o7yyottujl/df_current.csv?dl=0

df_b: https://www.dropbox.com/s/laveuldraurdpu1/df_climatology.csv?dl=0

Both of these have the same number and names of columns. However, when I do this:

pandas.concat([df_a, df_b]) 

I get the error:

AssertionError: Number of manager items must equal union of block items # manager items: 20, # tot_items: 21 

How to fix this?

Answer 1

I believe that this error occurs if the following two conditions are met:

1. The data frames have different columns. (i.e. (df1.columns == df2.columns) is False
2. The columns has a repeated value.

Basically if you concat dataframes with columns [A,B,C] and [B,C,D] it can work out to make one series for each distinct column name. So if I try to join a third dataframe [B,B,C] it does not know which column to append and ends up with fewer distinct columns than it thinks it needs.

If your dataframes are such that df1.columns == df2.columns then it will work anyway. So you can join [B,B,C] to [B,B,C], but not to [C,B,B], as if the columns are identical it probably just uses the integer indexes or something.