Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Explode an Array in Athena

I have a simple table in athena, it has an array of events. I want to write a simple select statement so that each event in array becomes a row.

I tried explode, transform, but no luck. I have successfully done it in Spark and Hive. But this Athena is tricking me. Please advise

DROP TABLE bi_data_lake.royalty_v4;
CREATE external TABLE bi_data_lake.royalty_v4 (
   KAFKA_ID string,
   KAFKA_TS string,
   deviceUser struct< deviceName:string, devicePlatform:string >,
   consumeReportingEvents array<
                                struct<
                                        consumeEvent: string,
                                        consumeEventAction: string,
                                        entryDateTime: string
                                      >
                               >
   )
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
LOCATION 's3://XXXXXXXXXXX';

Query which is not working

select kafka_id, kafka_ts,deviceuser, 
transform( consumereportingevents, consumereportingevent -> consumereportingevent.consumeevent) as cre
from bi_data_lake.royalty_v4 
where kafka_id = 'events-consumption-0-490565';

Not supported lateral view explode(consumereportingevents) as consumereportingevent

Answer to question it to use unnset

Found the answer for my question

WITH samples AS (
 select kafka_id, kafka_ts,deviceuser, consumereportingevent, consumereportingeventPos
 from bi_data_lake.royalty_v4 
 cross join unnest(consumereportingevents)  WITH ORDINALITY AS T (consumereportingevent, consumereportingeventPos)
 where kafka_id = 'events-consumption-0-490565' or kafka_id = 'events-consumption-0-490566'
)
SELECT * FROM samples
like image 974
Dave Avatar asked Sep 01 '25 02:09

Dave


1 Answers

Flatten ('explode') nested arrays in AWS Athena with UNNEST.

WITH dataset AS (
  SELECT
    'engineering' as department,
    ARRAY['Sharon', 'John', 'Bob', 'Sally'] as users
)
SELECT department, names FROM dataset
CROSS JOIN UNNEST(users) as t(names)

Reference: Flattening Nested Arrays

like image 94
pgzmnk Avatar answered Sep 03 '25 21:09

pgzmnk