Why does a conversion from np.datetime64 to float and back lead to a time difference?

Question

With the following code, I get a two hour difference after converting back to np.datetime64.

How can I avoid this? (if this should be a topic: I am presently in Central Europe)

import pandas as pd
import numpy as np
import datetime

a = np.datetime64('2018-04-01T15:30:00').astype("float")
a
b = np.datetime64(datetime.datetime.fromtimestamp(a))
b

Out[18]: numpy.datetime64('2018-04-01T17:30:00.000000')

Answer 1

The problem is not in the np.datetime64 conversion, but in datetime.datetime.fromtimestamp.

Since Numpy 1.11, np.datetime64 is timezone naive. It no longer assumes that input is in local time, nor does it print local times.

However, datetime.datetime.fromtimestamp does assume local time. From the docs:

Return the local date and time corresponding to the POSIX timestamp, such as is returned by time.time(). If optional argument tz is None or not specified, the timestamp is converted to the platform’s local date and time, and the returned datetime object is naive.

You can use datetime.datetime.utcfromtimestamp instead:

>>> a = np.datetime64('2018-04-01T15:30:00').astype("float")
>>> np.datetime64(datetime.datetime.utcfromtimestamp(a))
numpy.datetime64('2018-04-01T15:30:00.000000')

Answer 2

https://github.com/numpy/numpy/issues/3290

As of 1.7, datetime64 attempts to handle timezones by:

• Assuming all datetime64 objects are in UTC
• Applying timezone offsets when parsing ISO 8601 strings
• Applying the Locale timezone offset when the ISO string does not specify a TZ.
• Applying the Locale timezone offset when printing, etc.

https://stackoverflow.com/a/18817656/7583612

classmethod datetime.fromtimestamp(timestamp, tz=None)

Return the local date and time corresponding to the POSIX timestamp, such as is returned by time.time(). If optional argument tz is None or not specified, the timestamp is converted to the platform’s local date and time, and the returned datetime object is naive.

Else tz must be an instance of a class tzinfo subclass, and the timestamp is converted to tz‘s time zone. In this case the result is equivalent to tz.fromutc(datetime.utcfromtimestamp(timestamp).replace(tzinfo=tz))