Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Why is this datetime string not converting to a datetime object in a Pandas dataframe?

I am attempting to convert a column in a dataframe from an object to a datetime64[ns]. I am using to_datetime to accomplish this, yet at the end of my code, that column is still an object.

import pandas as pd
from StringIO import StringIO

DATA = StringIO("""id;Date of Event
3574;2015-12-12 22:03:28Z
0657;2015-08-25 17:48:03Z
0408;2015-10-13 12:01:32Z
3043;2015-09-08 16:55:43Z
9397;2015-09-09 09:33:31Z
9291;2015-07-15 08:13:48Z
4263;2015-12-30 09:25:55Z
0200;2015-10-25 13:38:35Z
8576;2015-09-01 02:01:47Z
6023;2015-08-29 20:47:20Z
9975;2015-10-05 15:11:32Z
5202;2015-12-21 23:44:10Z
9278;2015-12-22 05:56:03Z
8520;2015-09-05 01:27:07Z
9048;2015-08-29 18:38:26Z
9624;2015-12-09 01:49:15Z
2659;2015-10-03 01:43:50Z
6230;2015-10-16 11:43:40Z
2272;2015-11-18 14:15:52Z
""")

df = pd.DataFrame.from_csv(DATA, sep=";")
pd.to_datetime(df['Date of Event'], format="%Y-%m-%d %H:%M:%SZ")
print df['Date of Event'].dtype

That final print shows:

object

df.info() returns this:

Int64Index: 19 entries, 3574 to 2272
Data columns (total 1 columns):
Date of Event    19 non-null object
dtypes: object(1)
memory usage: 304.0+ bytes

Why did my pd.to_datetime(df['Date of Event'], format="%Y-%m-%d %H:%M:%SZ") fail to convert the column to datetime objects and how can I correct it?

The format is valid, and I can utilize the datetime library to test that:

>>> import datetime
>>> s = "2015-11-18 14:15:52Z"
>>> dt = datetime.datetime.strptime(s, "%Y-%m-%d %H:%M:%SZ")
>>> dt
datetime.datetime(2015, 11, 18, 14, 15, 52)

Why did the conversion fail on the entire Pandas column?

like image 740
Andy Avatar asked Mar 13 '23 14:03

Andy


1 Answers

to_datetime returns a new result, it doesn't modify its argument in place. Reassign it:

>>> df['Date of Event'] = pd.to_datetime(df['Date of Event'], format="%Y-%m-%d %H:%M:%SZ")
>>> df.dtypes
Date of Event    datetime64[ns] 
dtype: object

Or use parse_dates and have it converted at the start (note that it's more common to use read_csv than pd.DataFrame.from_csv):

>>> df = pd.read_csv(DATA, sep=";", parse_dates=["Date of Event"])
>>> df.dtypes
id                        int64
Date of Event    datetime64[ns]
dtype: object
like image 149
DSM Avatar answered Apr 27 '23 12:04

DSM