Pandas Groupby Consistent levels even if empty

Question

I'm trying to use group by to create a new dataframe, but I need the multi index to be consistent. Regardless of whether the sub category exists, I'd like it to be created like the following:

import pandas as pd

df = pd.DataFrame(
    {'Cat 1':['A','A','A','B','B','B','B','C','C','C','C','C','D'],
     'Cat 2':['A','B','A','B','B','B','A','B','B','B','B','B','A'],
     'Num':  [1,1,1,1,1,1,1,1,1,1,1,1,1]})
print df.groupby(['Cat 1','Cat 2']).sum()

With output that looks like:

             Num
Cat 1 Cat 2     
A     A        2
      B        1
B     A        1
      B        3
C     B        5
D     A        1

But I'd like it to look like

             Num
Cat 1 Cat 2     
A     A        2
      B        1
B     A        1
      B        3
C     A        Nan
      B        5
D     A        1
      B        Nan

I read different data that then adds a column in this format so the resulting array would look something like:

             Num        Num_added_later
Cat 1 Cat 2                
A     A        2         12
      B        1         5
B     A        1         5
      B        3         3
C     A        Nan       5
      B        5         5
D     A        1         1
      B        Nan       3

Answer 1

You can create a new index based on the two Cat columns and reindex your results:

import pandas as pd
new_index = pd.MultiIndex.from_product([df["Cat 1"].unique(), df["Cat 2"].unique()], names = ["Cat 1", "Cat 2"])

df.groupby(['Cat 1','Cat 2']).sum().reindex(new_index)

Answer 2

This is a hack1! Please use @Psidom's answer

df.groupby(['Cat 1','Cat 2']).sum().unstack().stack(dropna=False)

             Num
Cat 1 Cat 2     
A     A      2.0
      B      1.0
B     A      1.0
      B      3.0
C     A      NaN
      B      5.0
D     A      1.0
      B      NaN

---

Ok, maybe less a hack but...