PySpark - to_date format from column

Question

I am currently trying to figure out, how to pass the String - format argument to the to_date pyspark function via a column parameter.

Specifically, I have the following setup:

sc = SparkContext.getOrCreate()
df = sc.parallelize([('a','2018-01-01','yyyy-MM-dd'),
                      ('b','2018-02-02','yyyy-MM-dd'),
                      ('c','02-02-2018','dd-MM-yyyy')]).toDF(
                    ["col_name","value","format"])

I am currently trying to add a new column, where each of the dates from the column F.col("value"), which is a string value, is parsed to a date.

Separately for each format, this can be done with

df = df.withColumn("test1",F.to_date(F.col("value"),"yyyy-MM-dd")).\
        withColumn("test2",F.to_date(F.col("value"),"dd-MM-yyyy"))

This however gives me 2 new columns - but I want to have 1 column containing both results - but calling the column does not seem to be possible with the to_date function:

df = df.withColumn("test3",F.to_date(F.col("value"),F.col("format")))

Here an error "Column object not callable" is being thrown.

Is is possible to have a generic approach for all possible formats (so that I do not have to manually add new columns for each format)?

Answer 1

You can use a column value as a parameter without a udf using the spark-sql syntax:

Spark version 2.2 and above

from pyspark.sql.functions import expr
df.withColumn("test3",expr("to_date(value, format)")).show()
#+--------+----------+----------+----------+
#|col_name|     value|    format|     test3|
#+--------+----------+----------+----------+
#|       a|2018-01-01|yyyy-MM-dd|2018-01-01|
#|       b|2018-02-02|yyyy-MM-dd|2018-02-02|
#|       c|02-02-2018|dd-MM-yyyy|2018-02-02|
#+--------+----------+----------+----------+

Or equivalently using pyspark-sql:

df.createOrReplaceTempView("df")
spark.sql("select *, to_date(value, format) as test3 from df").show() 

Spark version 1.5 and above

Older versions of spark do not support having a format argument to the to_date function, so you'll have to use unix_timestamp and from_unixtime:

from pyspark.sql.functions import expr
df.withColumn(
    "test3",
    expr("from_unixtime(unix_timestamp(value,format))").cast("date")
).show()

Or equivalently using pyspark-sql:

df.createOrReplaceTempView("df")
spark.sql(
    "select *, cast(from_unixtime(unix_timestamp(value,format)) as date) as test3 from df"
).show()