Pandas pivot based on two columns (multi index)

Question

i have a DataFrame:

u_id    date        social_interaction_type_id  Total_Count
4       2018-08-19  4                           5
4       2018-08-24  2                           3
4       2018-08-21  1                           4

i want to pivot the DataFrame based on u_id and date.

so the result should look like something like this:

u_id    date        4       2       1
4       2018-08-19  5       nan     nan
4       2018-08-24  nan     3       nan
4       2018-08-21  nan     nan     4               

my code attempt:

df.pivot(index = ['u_id','date'] , columns='social_interaction_type_id',values='Total_Count')

error:

ValueError: Length of passed values is 8803, index implies 1

Answer 1

pd.DataFrame.pivot, for reasons unknown to me, don't work with a list of values for index. As per the docs, optional index must be a string or object. A workaround is to use pd.DataFrame.pivot_table with aggfunc='first':

res = df.pivot_table(index=['u_id', 'date'], columns='social_interaction_type_id',
                     values='Total_Count', aggfunc='first').reset_index()

print(res)

social_interaction_type_id  u_id        date    1    2    4
0                              4  2018-08-19  NaN  NaN  5.0
1                              4  2018-08-21  4.0  NaN  NaN
2                              4  2018-08-24  NaN  3.0  NaN

Answer 2

Use alternative solution with set_index and unstack:

df = (df.set_index(['u_id','date','social_interaction_type_id'])['Total_Count']
       .unstack()
       .reset_index()
       .rename_axis(None, axis=1))
print (df)
   u_id        date    1    2    4
0     4  2018-08-19  NaN  NaN  5.0
1     4  2018-08-21  4.0  NaN  NaN
2     4  2018-08-24  NaN  3.0  NaN

If duplicated in first 2 columns is necessary use aggregate function mean, sum like:

print (df)
   u_id        date  social_interaction_type_id  Total_Count
0     4  2018-08-19                           4            5 <- 4  2018-08-19 
1     4  2018-08-19                           6            4 <- 4  2018-08-19 
2     4  2018-08-24                           2            3
3     4  2018-08-21                           1            4


df2 = (df.groupby(['u_id','date','social_interaction_type_id'])['Total_Count']
       .mean() 
       .unstack()
       .reset_index()
       .rename_axis(None, axis=1))

Or:

df2 = (df.pivot_table(index=['u_id','date'],columns='social_interaction_type_id', values='Total_Count')
       .reset_index()
       .rename_axis(None, axis=1))

---

print (df2)
   u_id        date    1    2    4    6
0     4  2018-08-19  NaN  NaN  5.0  4.0
1     4  2018-08-21  4.0  NaN  NaN  NaN
2     4  2018-08-24  NaN  3.0  NaN  NaN