How to speed up DatetimeIndex processing?

Question

I have a large pandas DataFrame (around 1050000 entries). One of the columns is of type datetime. I want to extract year, month and weekday. The problem is that the code shown below is extremely slow:

df['Year'] = pd.DatetimeIndex(df.Date).year
df['Month'] = pd.DatetimeIndex(df.Date).month
df['Weekday'] = pd.DatetimeIndex(df.Date).weekday

Update:

The data looks like this:

      Id  DayOfWeek       Date
0      1          5 2015-07-31   
1      2          4 2015-07-30   
2      3          3 2015-07-29   
3      4          2 2015-07-28  
4      5          1 2015-07-27 

If I do this way:

df = pd.read_csv("data.csv", parse_dates=[2])

df['Year'] = pd.to_datetime(df['Date']).year
df['Month'] = pd.to_datetime(df['Date']).month
df['Weekday'] = pd.to_datetime(df['Date']).weekday

then the error is:

AttributeError: 'Series' object has no attribute 'year'

Answer 1

You state that your column is already of the datetime64 type. In that case you can simply use the .dt accessor to expose the methods and attributes associated with the datetime values in the column:

df['Year'] = df.Date.dt.year

This will be much quicker than writing pd.DatetimeIndex(df.Date).year which creates a whole new index object first.