While executing below query on Spark shell, I am facing partition error:
Expected only partition pruning predicates: ((((isnotnull(tenant_suite#478) && isnotnull(DS#477)) && (DS#477 >= 2017-06-01)) && (DS#477 <= 2017-06-25)) && (tenant_suite#478 = SAMS_CORESITE))
Not sure what error spark is throwing . Can anyone please help me with this?
SELECT
A.*
FROM
(-----------SUBQUERY 1
SELECT *
FROM
T2 -- PARTITION COLUMNS ARE DS AND TENANT_SUITE
WHERE
DS BETWEEN '2017-06-01'AND '2017-06-25'--date_sub(TO_DATE(FROM_UNIXTIME(UNIX_TIMESTAMP())),1)
AND tenant_suite = 'CORESITE'
) a
JOIN
( -----------SUBQUERY 1
SELECT
concat(concat(visid_high,'-',visid_low),'-',visit_num) AS VISIT_ID
,concat(visid_high,'-',visid_low) AS VISITOR_ID
,MAX(DS) AS EVENT_DT
FROM
T2 -- PARTITION COLUMNS ARE DS AND TENANT_SUITE
WHERE
tenant_suite = 'CORESITE'
AND DS BETWEEN '2017-06-01'AND '2017-06-25' --date_sub(TO_DATE(FROM_UNIXTIME(UNIX_TIMESTAMP())),1)
GROUP BY concat(concat(visid_high,'-',visid_low),'-',visit_num),concat(visid_high,'-',visid_low)
) B
ON A.VISIT_ID = B.VISIT_ID
AND A.VISITOR_ID = B.VISITOR_ID
AND A.VISIT_DT = B.EVENT_DT
group by a.VISIT_DT;
In Spark Partition Columns are case sensitive- Spark Jira Issue
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With