Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

convert epoch to datetime in Scala / Spark

I'm converting String representing a DateTime to unix_time (epoch) using :

def strToTime(x: String):Long = { DateTimeFormat.
    forPattern("YYYY-MM-dd HH:mm:ss").parseDateTime(x).getMillis()/1000 }

to get a list of Long like this :

.map( p=> List( strToTime(p(0) ) ) ) 

my question is - what is the easiest way to turn in backwards? something like:

def timeToStr(x: Long):String = { x*1000L.toDateTime}

that I could use on the above List(Long)

I have read Convert seconds since epoch to joda DateTime in Scala but can't apply it successfully

like image 510
Zahiro Mor Avatar asked Nov 02 '15 10:11

Zahiro Mor


People also ask

How do I change my epoch time to date on Spark?

Spark SQL Function from_unixtime() is used to convert the Unix timestamp to a String representing Date and Timestamp, in other words, it converts the Epoch time in seconds to date and timestamp.

How to convert Epoch time to timestamp in Spark?

from_unixtime() SQL function is used to convert or cast Epoch time to timestamp string and this function takes Epoch time as a first argument and formatted string time as the second argument. As a first argument, we use unix_timestamp() which returns the current timestamp in Epoch time (Long) as an argument.

What is epoch in spark?

For usability, Spark SQL recognizes special string values in all methods that accept a string and return a timestamp or date: epoch is an alias for date 1970-01-01 or timestamp 1970-01-01 00:00:00Z .

How do I change time to date in epoch shell script?

How to convert seconds since the epoch (1970-01-01 UTC) to a date in Linux? You can use the date command on Linux to convert the time formats. You can control the format of the output by add +FORMAT at the end of the command. For detailed introduction to the FORMAT string, check the manual of date .


3 Answers

Follows my approach!

import java.util.Date
import java.text.SimpleDateFormat

def epochToDate(epochMillis: Long): String = {
    val df:SimpleDateFormat = new SimpleDateFormat("yyyy-MM-dd")
    df.format(epochMillis)
}   

Follows a test run.

scala> epochToDate(1515027919000L)
res0: String = 2018-01-03
like image 127
Rahul Avatar answered Oct 22 '22 19:10

Rahul


You have a precedence problem - .toDateTime is being applied to 1000L before * is applied. Bracket the operations to make the call order clear:

def timeToStr(x: Long): String = { (x*1000L).toDateTime }
like image 21
Shadowlands Avatar answered Oct 22 '22 19:10

Shadowlands


The opposite of parseDateTime is print :)

def timeToStr(epochMillis: Long): String =
  DateTimeFormat.forPattern("YYYY-MM-dd HH:mm:ss").print(epochMillis)
like image 6
Zoltán Avatar answered Oct 22 '22 20:10

Zoltán