pandas - concat with columns of same categories turns to object

Question

I want to concatenate two dataframes with category-type columns, by first adding the missing categories to each column.

df = pd.DataFrame({"a": pd.Categorical(["foo", "foo", "bar"]), "b": [1, 2, 1]})
df2 = pd.DataFrame({"a": pd.Categorical(["baz"]), "b": [1]})

df["a"] = df["a"].cat.add_categories("baz")
df2["a"] = df2["a"].cat.add_categories(["foo", "bar"])

In theory the categories for both "a" columns are the same:

In [33]: df.a.cat.categories
Out[33]: Index(['bar', 'foo', 'baz'], dtype='object')

In [34]: df2.a.cat.categories
Out[34]: Index(['baz', 'foo', 'bar'], dtype='object')

However, when concatenating the two dataframes, I get an object-type "a" column:

In [35]: pd.concat([df, df2]).info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 4 entries, 0 to 0
Data columns (total 2 columns):
a    4 non-null object
b    4 non-null int64
dtypes: int64(1), object(1)
memory usage: 96.0+ bytes

In the documentation it says that when categories are the same, it should result in a category-type column. Does the order of the categories matter even though the category is unordered? I am using pandas-0.20.3.

Answer 1

Yes. By using reorder_categories you can change the order of categories, even though the category itself is unordered.

df2["a"] = df2.a.cat.reorder_categories(df.a.cat.categories)

In [43]: pd.concat([df, df2]).info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 4 entries, 0 to 0
Data columns (total 2 columns):
a    4 non-null category
b    4 non-null int64
dtypes: category(1), int64(1)
memory usage: 172.0 bytes