Menu
I am trying to optimize a very long and complex impala query which contains multiple CTE. Each CTE is used multiple times. My expectation is that once a CTE is created, I should be able to direct impala that results of this CTE should be re-used in main query as-is instead of SCAN HDFS operation on tables involved in CTE again with the main query. Is this possible? if yes how ?
I am using impalad version 2.1.1-cdh5 RELEASE (build 7901877736e29716147c4804b0841afc4ebc9037) version
329
There's currently no test to diagnose CTE. A diagnosis is based on a history of participating in contact sports, plus the symptoms and clinical features. Your GP will talk to you about the problems you're experiencing and may ask you to carry out some simple mental or physical tasks, such as moving or walking around.
The symptoms of CTE generally do not present until years or decades after the brain trauma occurred or after one stops actively playing contact sports.
While chronic traumatic encephalopathy (CTE) cannot yet be diagnosed during life, a new study provides the best evidence to date that a commonly used brain imaging technique, magnetic resonance imaging (MRI), may expedite the ability to diagnose CTE with confidence in the living.
CTE symptoms: Development and detection. Stage 1: Short-term memory loss; mild aggression and depression; headaches. Stage 2: Severe depression, outbursts, and mood swings.
I do not think so. I believe WITH clause does not create any permanent object, it's just for you avoid cluttering the namespace with new tables or views and to make it easier to refactor large, complex queries by reor‐ dering and replacing their individual parts. The queries used in the WITH clause are good candidates to someday become views or to be materialized as summary tables during the ETL process.
152
Is this possible?
The very purpose of the CTE is to re-use of the results obtained from a preceding query (that uses the with clause) by a following query, say, SELECT. So i don't see a reason why it is not possible.
Use Explain on your query to find out the actual SCAN HDFS details.
For more I/O related insights use profile as in the official documentation https://www.cloudera.com/documentation/enterprise/5-7-x/topics/impala_explain_plan.html#perf_profile
36
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
