Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Why does a query fail with "AnalysisException: Expected only partition pruning predicates"?

While executing below query on Spark shell, I am facing partition error:

Expected only partition pruning predicates: ((((isnotnull(tenant_suite#478) && isnotnull(DS#477)) && (DS#477 >= 2017-06-01)) && (DS#477 <= 2017-06-25)) && (tenant_suite#478 = SAMS_CORESITE))

Not sure what error spark is throwing . Can anyone please help me with this?

SELECT 
  A.*
FROM
        (-----------SUBQUERY 1
        SELECT * 
        FROM 
            T2 --  PARTITION COLUMNS ARE DS AND TENANT_SUITE
        WHERE   
            DS BETWEEN '2017-06-01'AND '2017-06-25'--date_sub(TO_DATE(FROM_UNIXTIME(UNIX_TIMESTAMP())),1)
            AND tenant_suite = 'CORESITE'
        ) a

        JOIN

        ( -----------SUBQUERY 1
        SELECT
             concat(concat(visid_high,'-',visid_low),'-',visit_num) AS VISIT_ID
            ,concat(visid_high,'-',visid_low) AS VISITOR_ID
            ,MAX(DS) AS EVENT_DT
        FROM
            T2  -- PARTITION COLUMNS ARE DS AND TENANT_SUITE
        WHERE
            tenant_suite = 'CORESITE'
            AND DS BETWEEN '2017-06-01'AND '2017-06-25' --date_sub(TO_DATE(FROM_UNIXTIME(UNIX_TIMESTAMP())),1) 
        GROUP BY concat(concat(visid_high,'-',visid_low),'-',visit_num),concat(visid_high,'-',visid_low)
        ) B
            ON A.VISIT_ID = B.VISIT_ID
            AND A.VISITOR_ID = B.VISITOR_ID
            AND A.VISIT_DT = B.EVENT_DT
    group by a.VISIT_DT;
like image 941
Umamaheshwar Avatar asked Jul 11 '17 01:07

Umamaheshwar


1 Answers

In Spark Partition Columns are case sensitive- Spark Jira Issue

like image 81
shivMagus Avatar answered Sep 27 '22 22:09

shivMagus