I have several Dataframes with the same columns that I'd like to merge on their indices only.
print df1
out[]: Value ISO
Id
200001 8432000000 USD
200230 22588186000 USD
200247 4633000000 USD
200291 1188880000 USD
200418 1779776000 USD
print df2
out[]: Value ISO
Id
200001 1.309168e+11 USD
200230 5.444096e+10 USD
200247 9.499602e+09 USD
200291 2.089603e+09 USD
200418 3.827251e+09 USD
print df3
out[]: Value
Id
200001 3.681908
200230 3.408507
200247 4.531866
200291 0.273029
200418 3.521822
I could use
pd.concat([df1, df2, df3], axis=1)
and get
out[]: Value ISO Value ISO Value
Id
200001 8432000000 USD 1.309168e+11 USD 3.681908
200230 22588186000 USD 5.444096e+10 USD 3.408507
200247 4633000000 USD 9.499602e+09 USD 4.531866
200291 1188880000 USD 2.089603e+09 USD 0.273029
200418 1779776000 USD 3.827251e+09 USD 3.521822
But I lose the information of where each column came from. I could also do a merge on two dataframes and use the suffixes parameter
print df1.merge(df2, left_index=True, right_index=True, suffixes=('_1', '_2'))
and get
out[]: Value_1 ISO_1 Value_2 ISO_2
Id
200001 8432000000 USD 1.309168e+11 USD
200230 22588186000 USD 5.444096e+10 USD
200247 4633000000 USD 9.499602e+09 USD
200291 1188880000 USD 2.089603e+09 USD
200418 1779776000 USD 3.827251e+09 USD
I can then daisy chain my merges but the suffixes parameter only applies to columns that share a name. Once I've suffixed the first merge, the names will no longer be in common with the third dataframe.
I figured the solution would be to append a level to the column index of each dataframe with the relevant information necessary to distinguish those columns. Then I could run a pd.concat() and get something that looks like this:
print pd.concat([df1_, df2_, df3_], axis=1)
out[]:Source df1 df2 df3
Value ISO Value ISO Value
200001 8.432e+09 USD 1.309168e+11 USD 3.681908
200230 2.258819e+10 USD 5.444096e+10 USD 3.408507
200247 4.633e+09 USD 9.499602e+09 USD 4.531866
200291 1.18888e+09 USD 2.089603e+09 USD 0.273029
200418 1.779776e+09 USD 3.827251e+09 USD 3.521822
However, in order to get this to happen. I had to abuse the dataframes like so:
df1_ = df1.T
df1_['Source'] = 'df1'
df1_.set_index('Source', append=True, inplace=True)
df1_.index = df1_.index.swaplevel(0, 1)
df1_ = df1_.T
Ultimately, I want a result to look a lot like the last concat statement. Is there a better way to get there? Is there a better way to append a level to the column index?
Thanks, PiR
I you want a MultiIndex, you can do this directly in the concat
function to get the same results, like:
pd.concat([df1, df2, df3], axis=1, keys=['df1', 'df2', 'df3'])
or
pd.concat({'df1':df1, 'df2':df2, 'df3':df3}, axis=1)
See also Multi-index dataframe from sequence of dataframes
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With