How to convert pandas dataframe columns to native python data types?

Question

I have a dataframe whose columns data types need to be mapped to python native data types.

I want to be able to get a dictionary from numpy and convert each column to it's native type.

for example:

{numpy.object_: object,
 numpy.bool_: bool,
 numpy.string_: str,
 numpy.unicode_: unicode,
 numpy.int64: int,
 numpy.float64: float,
 numpy.complex128: complex}

I tried both astype and pd.to_numeric, neither downcasts the column sufficiently.

df['source'] = df['source'].astype(int) returns int32, as does pd.to_numeric

Update:

Most of the comments question the wisdom for doing this. networkx reads dataframes and accepts np datatypes. However the graph cannot be written using json_dumps because of this well documented error: TypeError: Object of type 'int64' is not JSON serializable

Thanks

Answer 1

"Native Python type" to pandas (or to numpy) is an object. That's the extent of it. Pandas only knows it's a Python object and act accordingly. Other than that, you cannot have columns of type string, unicode, integers etc.

You can have object columns and store whatever you want inside them, though. Pandas will handle most of the conversion for you at this stage.

df = pd.DataFrame({'A': [1, 2], 
                   'B': [1., 2.], 
                   'C': [1 + 2j, 3 + 4j], 
                   'D': [True, False], 
                   'E': ['a', 'b'], 
                   'F': [b'a', b'b']})

df.dtypes
Out[71]: 
A         int64
B       float64
C    complex128
D          bool
E        object
F        object
dtype: object

for col in df:
    print(type(df.loc[0, col]))

<class 'numpy.int64'>
<class 'numpy.float64'>
<class 'numpy.complex128'>
<class 'numpy.bool_'>
<class 'str'>
<class 'bytes'>

---

df = df.astype('object')

for col in df:
    print(type(df.loc[0, col]))

<class 'int'>
<class 'float'>
<class 'complex'>
<class 'bool'>
<class 'str'>
<class 'bytes'>