Menu
How to get current_date - 1 day in sparksql, same as cur_date()-1 in mysql.
766
In order to populate current date and current timestamp in pyspark we will be using current_date() and current_timestamp() function respectively. current_date() function populates current date in a column in pyspark.
current_timestamp(): This function returns the current timestamp in the apache spark. The default format produced is in yyyy-MM-dd HH:mm:ss. The Spark SQL functions package is imported into the environment to run Timestamp functions. dataframe value is created which is used to store seq() funtion.
Spark SQL provides DataFrame function add_months() to add or subtract months from a Date Column and date_add() , date_sub() to add and subtract days. Below code, add days and months to Dataframe column, when the input Date in “yyyy-MM-dd” Spark DateType format.
The arithmetic functions allow you to perform arithmetic operation on columns containing dates.
For example, you can calculate the difference between two dates, add days to a date, or subtract days from a date. The built-in date arithmetic functions include datediff, date_add, date_sub, add_months, last_day, next_day, and months_between.
Out of above what we need is
date_sub(timestamp startdate, int days), Purpose: Subtracts a specified number of days from a TIMESTAMP value. The first argument can be a string, which is automatically cast to TIMESTAMP if it uses the recognized format, as described in TIMESTAMP Data Type. Return type: Returns the date that is > days days before start
and we have
current_timestamp() Purpose: Alias for the now() function. Return type: timestamp
you can do select
date_sub(CAST(current_timestamp() as DATE), 1) See https://spark.apache.org/docs/1.6.2/api/java/org/apache/spark/sql/functions.html
126
You can try
date_add(current_date(), -1) I don't know spark either but I found it on google. You can also use this link for reference
42
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
