Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Get mapping of categorical variables in pandas

Tags:

python

pandas

I'm doing this to make categorical variables numbers

>>> df = pd.DataFrame({'x':['good', 'bad', 'good', 'great']}, dtype='category')         x 0   good 1    bad 2   good 3  great 

How can I get the mapping between the original values and the new values?

like image 461
Bob Avatar asked May 28 '15 15:05

Bob


People also ask

How do you summarize categorical data in pandas?

Proportions are often used to summarize categorical data and can be calculated by dividing individual frequencies by the total number of responses. In Python/pandas, df['column_name']. value_counts(normalize=True) will ignore missing data and divide the frequency of each category by the total in any category.

How do you filter categorical data in pandas?

For categorical data you can use Pandas string functions to filter the data. The startswith() function returns rows where a given column contains values that start with a certain value, and endswith() which returns rows with values that end with a certain value.


1 Answers

Method 1

You can create a dictionary mapping by enumerating (similar to creating a dictionary from a list by creating dictionary keys from the list indices):

dict( enumerate(df['x'].cat.categories ) )  # {0: 'bad', 1: 'good', 2: 'great'} 

Method 2

Alternatively, you could map the values and codes in every row:

dict( zip( df['x'].cat.codes, df['x'] ) )  # {0: 'bad', 1: 'good', 2: 'great'} 

It's a little more transparent what is happening here, and arguably safer for that reason. It is also much less efficient as the length of the arguments to zip() is len(df) whereas the length of df['x'].cat.categories is only the count of unique values and generally much shorter than len(df).

Additional Discussion

The reason Method 1 works is that the categories have type Index:

type( df['x'].cat.categories )  # pandas.core.indexes.base.Index 

and in this case you look up values in an index just as you would a list.

There are a couple of ways to verify that Method 1 works. First, you can just check that a round trip retains the correct values:

(df['x'] == df['x'].cat.codes.map( dict(              enumerate(df['x'].cat.categories) ) ).astype('category')).all() # True 

or you can check that Method 1 and Method 2 give the same answer:

(dict( enumerate(df['x'].cat.categories ) ) == dict( zip( df['x'].cat.codes, df['x'] ) ))  # True 
like image 72
JohnE Avatar answered Oct 01 '22 12:10

JohnE