Pandas Datetime Conversion from a particular ISO format

Question

Many thanks for your help in advance.

I'm trying to convert a datetime which is a string in ISO Format to a datetime object. But have tried many methods without being successful. Please your help with this.

As an example, I have a dataframe which the column time is similar as shown below. This was extracted from a database and this was the output's format.

2018-12-04T04:39:26Z
2018-12-04T05:10:54.6Z
2018-12-04T05:17:32Z
2018-12-04T10:51:20.5Z
...

What I have tried so far (many attempts) but not being successful:

df.index = pd.to_datetime(df.index, format = "%Y-%m-%dT%H:%M:%SZ", errors='ignore')

df.index = pd.to_datetime(df.index)

df.time = df.time.map(lambda x: pd.to_datetime(dt.datetime.strptime(x, '%Y-%m-%dT%H:%M:%SZ'), format = '%d/%m/%Y %H:%M'))

Thanks Again!

Answer 1

I wanted to answer this before. In the end I just made a function that deals with different data inputs and also creates a dataframe with column names. Thanks ALollz for your comment regarding pd.to_datetime(df.index, errors='coerce').

So to convert the index from a string in an ISO format I established and followed this sequence:

df = pd.DataFrame([[-1.8, '2018-09-14T13:36:00Z']], columns = ['current', 'time'])
df.set_index('time', inplace = True)   # make it your index by using the inplace=True
df.index = pd.to_datetime(df.index, errors='coerce')

After the conversion to datetime, check that dates are correct. You may need to specify the format if they are wrong read.

Thanks!

Answer 2

A bit late to the party, but I believe that this response needs to be visible to ease people's lives.

If, as you said, it was extracted from a database, then you can do it directly at the moment of establishing your dataframe. Most pandas read functions have a parameter parse_dates. As said in the documentation:

Note: A fast-path exists for iso8601-formatted dates.

So even if you have 2 or more columns with dates, you can do it in an extremely easy manner.

df = pd.read_csv("x.csv", parse_dates=['Date1', "Date2"], names=["ID", "Date1", "Date2"])