Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Pandas invalid type comparison error

For some reason, which I can't find in the Pandas changelog for 0.17.1, comparing a datetime series with an int value (Unix epoch) does not work anymore. Could anyone please explain this or point me to the right section in the changelog?

Working in 0.16.2

>>> import pandas as pd
>>> import datetime
>>> d = pd.Series([datetime.datetime(2016, 1, 1), datetime.datetime(2016, 1, 1)])
>>> d
0   2016-01-01
1   2016-01-01
dtype: datetime64[ns]
>>> d.dtype
dtype('<M8[ns]')
>>> d > 10
0    True
1    True
dtype: bool

Error in 0.17.1

>>> import pandas as pd
>>> import datetime
>>> d = pd.Series([datetime.datetime(2016, 1, 1), datetime.datetime(2016, 1, 1)])
>>> d > 10
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/sven/tmp/pandastest/pandas-0.17.1/lib/python2.7/site-packages/pandas/core/ops.py", line 726, in wrapper
    res = na_op(values, other)
  File "/Users/sven/tmp/pandastest/pandas-0.17.1/lib/python2.7/site-packages/pandas/core/ops.py", line 657, in na_op
    raise TypeError("invalid type comparison")
TypeError: invalid type comparison
like image 633
orange Avatar asked Feb 08 '16 08:02

orange


1 Answers

You can still use an explicit conversion:

u_time_ns = d.apply(lambda x: x.to_datetime64().view('int64'))
u_time_ns

0    1451606400000000000
1    1451606400000000000
dtype: int64

u_time_ns > 10

0    True
1    True
dtype: bool

Or, if you want to rely on pandas timestamps being stored as datetime64[ns]:

u_time_ns = d.view('int64')

Sorry, no idea why this changed.

like image 152
Stop harming Monica Avatar answered Nov 01 '22 18:11

Stop harming Monica