How to convert a Pandas data frame column from np.datetime64 to datetime?

Question

I would like to put a Pandas Data Frame column into datetime format from datetime64. This works on an an individual basis. In particular the following works fine:

t = dt['time'].values[0]
datetime.utcfromtimestamp(t.astype(int)/1000000000)

However, when I try to do this to the entire column

dt['datetime'] = dt['time'].apply(lambda x: datetime.utcfromtimestamp(x.astype(int)/1000000000))

I get the following error:

pandas/src/inference.pyx in pandas.lib.map_infer (pandas/lib.c:62578)()

<ipython-input-26-5950d82979b4> in <lambda>(x)
      1 print(type(dt['time'].values[0]))
      2 
----> 3 dt['datetime'] = dt['time'].apply(lambda x: datetime.utcfromtimestamp(x.astype(int)/1000000000))
      4 t = dt['time'].values[0]
      5 print(t)

AttributeError: 'Timestamp' object has no attribute 'astype'

What am I doing wrong? How can I convert my column to datetime and/or make a new column in datetime format?

Here is the info for the dataframe:

Answer 1

You can convert Series of dtype datetime64[ns] to a NumPy array of datetime.datetime objects by calling the .dt.to_pydatetime() method:

In [75]: df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 252 entries, 0 to 251
Data columns (total 1 columns):
time    252 non-null datetime64[ns]<--the `time` column has dtype `datetime64[ns]`
dtypes: datetime64[ns](1)
memory usage: 2.0 KB

In [77]: df.head()
Out[77]: 
        time
0 2009-01-02
1 2009-01-05
2 2009-01-06
3 2009-01-07
4 2009-01-08


In [76]: df['time'].dt.to_pydatetime()[:5]
Out[76]: 
array([datetime.datetime(2009, 1, 2, 0, 0),
       datetime.datetime(2009, 1, 5, 0, 0),
       datetime.datetime(2009, 1, 6, 0, 0),
       datetime.datetime(2009, 1, 7, 0, 0),
       datetime.datetime(2009, 1, 8, 0, 0)], dtype=object)

---

Note that NDFrames (such as Series and DataFrames) can only hold datetime-like objects as objects of dtype datetime64[ns]. The automatic conversion of all datetime-likes to a common dtype simplifies subsequent date computations. But it makes it impossible to store, say, Python datetime.datetime objects in a DataFrame column. Pandas core developer, Jeff Reback explains,

"We don't allow direct conversions because its simply too complicated to keep anything other than datetime64[ns] internally (nor necessary at all)."