Menu
I have a DataFrame with Timestamp column, which i need to convert as Date format.
Is there any Spark SQL functions available for this?
626
You can simply use the fromtimestamp function from the DateTime module to get a date from a UNIX timestamp. This function takes the timestamp as input and returns the corresponding DateTime object to timestamp.
read_csv(), the timestamps column from the data Dataframe is given as an argument in the to_datetime() for it to be converted into DateTime. unit='s' is used to convert the values of the timestamp column to epoch time after converting the values to DateTime it is stored in a column called 'Datetime' in the Dataframe.
Function usedstrftime() can change the date format in python.
You can cast the column to date:
Scala:
import org.apache.spark.sql.types.DateType val newDF = df.withColumn("dateColumn", df("timestampColumn").cast(DateType)) Pyspark:
df = df.withColumn('dateColumn', df['timestampColumn'].cast('date')) In SparkSQL:
SELECT CAST(the_ts AS DATE) AS the_date FROM the_table If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
