How to translate "bytes" objects into literal strings in pandas Dataframe, Python3.x?

Question

I have a Python3.x pandas DataFrame whereby certain columns are strings which as expressed as bytes (like in Python2.x)

import pandas as pd df = pd.DataFrame(...) df        COLUMN1         .... 0      b'abcde'        .... 1      b'dog'          .... 2      b'cat1'         .... 3      b'bird1'        .... 4      b'elephant1'    .... 

When I access by column with df.COLUMN1, I see Name: COLUMN1, dtype: object

However, if I access by element, it is a "bytes" object

df.COLUMN1.ix[0].dtype Traceback (most recent call last):   File "<stdin>", line 1, in <module> AttributeError: 'bytes' object has no attribute 'dtype' 

How do I convert these into "regular" strings? That is, how can I get rid of this b'' prefix?

Answer 1

You can use vectorised str.decode to decode byte strings into ordinary strings:

df['COLUMN1'].str.decode("utf-8") 

To do this for multiple columns you can select just the str columns:

str_df = df.select_dtypes([np.object]) 

convert all of them:

str_df = str_df.stack().str.decode('utf-8').unstack() 

You can then swap out converted cols with the original df cols:

for col in str_df:     df[col] = str_df[col]