Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Convert epoch, which is midnight 01/01/0001, to DateTime in pandas

Tags:

python

pandas

I have a column with a series of timestamps in it. Originally I thought they are in Unix timestamps system so I used the following code to convert them to date time.

big_frame['date'] = pd.to_datetime(big_frame['filename'],unit='s')

however, it gave me odd results so I researched a bit more and found out the timestamps are basically using the .net epoch, which is midnight 01/01/0001. So the file names are the number of seconds from that epoch. How can I convert a column in my data frame to the right DateTime?

For example if I have this

63730342900 

The result should be

14/07/2020 17:01:40

Edit:

https://www.epochconverter.com/seconds-days-since-y0

This is the only site I could find that converts the above timestamp correctly

Below is the column I want to convert

0     63729045145
1     63729045145
2     63729045146
3     63729045146
4     63729045146
5     63729045147
6     63729045147
7     63729045147
like image 394
hyeri Avatar asked Jul 21 '20 12:07

hyeri


People also ask

How do I convert epoch to date?

Convert from epoch to human-readable dateString date = new java.text.SimpleDateFormat("MM/dd/yyyy HH:mm:ss").format(new java.util.Date (epoch*1000)); Epoch in seconds, remove '*1000' for milliseconds. myString := DateTimeToStr(UnixToDateTime(Epoch)); Where Epoch is a signed integer.

How do I convert epoch time to manual date?

You can take an epoch time divided by 86400 (seconds in a day) floored and add 719163 (the days up to the year 1970) to pass to it. Awesome, this is as manual as it gets.


1 Answers

01/01/0001 seems to be out of range for datetime/timedelta type. We could do a little hack like this:

ref_date = pd.to_datetime('14/07/2020 17:01:40')
ref_stamp = 63730342900


bigframe['date'] = pd.to_timedelta(big_frame['date'] - ref_stamp, unit='s') + ref_date

Output:

0   2020-06-29 16:32:25
1   2020-06-29 16:32:25
2   2020-06-29 16:32:26
3   2020-06-29 16:32:26
4   2020-06-29 16:32:26
5   2020-06-29 16:32:27
6   2020-06-29 16:32:27
7   2020-06-29 16:32:27
Name: date, dtype: datetime64[ns]
like image 74
Quang Hoang Avatar answered Oct 04 '22 01:10

Quang Hoang