Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

pandas read_csv column dtype is set to decimal but converts to string

I am using pandas (v0.18.1) to import the following data from a file called 'test.csv':

a,b,c,d
1,1,1,1.0

I have set the dtype to 'decimal.Decimal' for columns 'c' and 'd' but instead they return as type 'str'.

import pandas as pd
import decimal as D

df = pd.read_csv('test.csv', dtype={'a': int, 'b': float, 'c': D.Decimal, 'd': D.Decimal})

for i, v in df.iterrows():
    print(type(v.a), type(v.b), type(v.c), type(v.d))

Results:

`<class 'int'> <class 'float'> <class 'str'> <class 'str'>`

I have also tried converting to decimal explicitly after import with no luck (converting to float works but not decimal).

df.c = df.c.astype(float)
df.d = df.d.astype(D.Decimal)
for i, v in df.iterrows():
    print(type(v.a), type(v.b), type(v.c), type(v.d))

Results:

`<class 'int'> <class 'float'> <class 'float'> <class 'str'>`

The following code converts a 'str' to 'decimal.Decimal' so I don't understand why pandas doesn't behave the same way.

x = D.Decimal('1.0')
print(type(x))

Results:

`<class 'decimal.Decimal'>`
like image 561
candleford Avatar asked Jun 30 '16 05:06

candleford


People also ask

Does pandas have decimal type?

Pandas can use Decimal, but requires some care to create and maintain Decimal objects.

What output type does pandas read_csv () return?

Read a CSV File In this case, the Pandas read_csv() function returns a new DataFrame with the data and labels from the file data. csv , which you specified with the first argument. This string can be any valid path, including URLs.

How do I change the datatype in pandas?

Change data type of a series in PandasUse a numpy. dtype or Python type to cast entire pandas object to the same type. Alternatively, use {col: dtype, …}, where col is a column label and dtype is a numpy. dtype or Python type to cast one or more of the DataFrame's columns to column-specific types.


1 Answers

I think you need converters:

import pandas as pd
import io
import decimal as D

temp = u"""a,b,c,d
           1,1,1,1.0"""

# after testing replace io.StringIO(temp) to filename
df = pd.read_csv(io.StringIO(temp), 
                 dtype={'a': int, 'b': float}, 
                 converters={'c': D.Decimal, 'd': D.Decimal})

print (df)
       a    b  c    d
    0  1  1.0  1  1.0

for i, v in df.iterrows():
    print(type(v.a), type(v.b), type(v.c), type(v.d))

    <class 'int'> <class 'float'> <class 'decimal.Decimal'> <class 'decimal.Decimal'>
like image 163
jezrael Avatar answered Oct 05 '22 23:10

jezrael