Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How can I check the partition list from Athena in AWS?

I want to check the partition lists in Athena.

I used query like this.

show partitions table_name

But I want to search specific table existed.

So I used query like below but there was no results returned.

show partitions table_name partition(dt='2010-03-03')

Because dt contains hour data also.

dt='2010-03-03-01', dt='2010-03-03-02', ...........

So is there any way to search when I input '2010-03-03' then it search '2010-03-03-01', '2010-03-03-02'?

Do I have to separate partition like this?

dt='2010-03-03', dh='01'

And show partitions table_name returned only 500 rows in Hive. Is the same in Athena also?

like image 571
Bethlee Avatar asked May 24 '17 01:05

Bethlee


People also ask

How many partitions can an Athena table have?

Although Athena supports querying AWS Glue tables that have 10 million partitions, Athena cannot read more than 1 million partitions in a single scan. In such scenarios, partition indexing can be beneficial.

How do I check data Athena?

In the Athena console, choose Views, and then choose a view to expand it and show the columns in the view.

How do I know the size of my Athena table?

The quick way is via s3: ... > Show Properties > Location and lookup the size in the s3-console. You can run SELECT * FROM some_table for each table and look at the result metadata for the amount scanned, but it will be an expensive way to do it.


1 Answers

In Athena v2:

Use this SQL:

SELECT dt
FROM db_name."table_name$partitions"
WHERE dt LIKE '2010-03-03-%'

(see the official aws docs)


In Athena v1:

There is a way to return the partition list as a resultset, so this can be filtered using LIKE. But you need to use the internal information_schema database like this:

SELECT partition_value
FROM information_schema.__internal_partitions__
WHERE table_schema = '<DB_NAME>'
        AND table_name = '<TABLE_NAME>'
        AND partition_value LIKE '2010-03-03-%'
like image 187
Mariusz Avatar answered Sep 21 '22 23:09

Mariusz