Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Hive describe partitions to show partition url

I know that there is

DESCRIBE FORMATTED table_name;

that shows you the table format. Is there a way to get a more information about partitions apart from

SHOW PARTITIONS table_name;

I saw that Hive language manual has this

DESCRIBE [EXTENDED|FORMATTED] [db_name.]table_name PARTITION partition_spec

I would like to view all the partitions along with the url in hdfs or s3 where the data is stored.

like image 993
viper Avatar asked Feb 27 '14 00:02

viper


People also ask

How do I find partition details in Hive?

Show All Partitions on Hive Table. After loading the data into the Hive partition table, you can use SHOW PARTITIONS command to see all partitions that are present. Alternatively, if you know the Hive store location on the HDFS for your table, you can run the HDFS command to check the partitions.

How do I query the default Hive partition?

The HIVE_DEFAULT_PARTITION in hive is represented by a NULL value of the partitioned column. That means, if we have a NULL value for a partition column and loading this record to a partitioned table, then hive_default_partition will get create for that record.

How do I select a partition column in Hive?

Static Partitioning in Hive While loading data, you need to specify which partition to store the data in. This means that with each load, you need to specify the partition column value. You can add a partition in the table and move the data file into the partition of the table.

How can I find the partition of a table?

All tables have at least one partition, so if you are looking specifically for partitioned tables, then you'll have to filter this query based off of sys. partitions. partition_number <> 1 (for non-partitioned tables, the partition_number is always equal to 1).


2 Answers

To show partitions:

show partitions table_name

To show where a partition is physically stored:

describe formatted dbname.tablename partition (name=value)

I don't know of a built-in way to create an output that is (partition, path) but you can build it using these two commands and some grep/awk or whatever.

like image 121
Jon Watte Avatar answered Oct 19 '22 04:10

Jon Watte


analyze table TABLENAME partition(the_partition) compute statistics nopass;

The above code gives you more info about the partitions (number of files, number of rows, total size), but doesn't give you exact location.

If you want exact location, you may want to create an external table.

like image 24
JJFord3 Avatar answered Oct 19 '22 06:10

JJFord3