Checking for date validity in spark sql

Question

Is it possible to check if a date(yyymmdd) is valid in spark SQL? I have tried the following:

TO_DATE(CAST(UNIX_TIMESTAMP(CAST(UNIX_TIMESTAMP(colname, 'yyyymmdd') AS  TIMESTAMP))) IS NOT NULL

Thanks, Bharath

Answer 1

You could try

set spark.sql.legacy.timeParserPolicy=CORRECTED;

before the select. This policy allows invalid dates to be casted as null.

Hope it helps.