Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to Update/Drop a Hive Partition?

Tags:

hive

hiveql

After adding a partition to an external table in Hive, how can I update/drop it?

like image 463
darcyy Avatar asked Dec 11 '12 07:12

darcyy


People also ask

How do I update my Hive partition?

You can refresh Hive metastore partition information manually or automatically. You run the MSCK (metastore consistency check) Hive command: MSCK REPAIR TABLE table_name SYNC PARTITIONS every time you need to synchronize a partition with your file system. You set up partition discovery to occur periodically.

How do I drop a specific partition in Hive?

Drop or Delete Hive Partition You can use ALTER TABLE with DROP PARTITION option to drop a partition for a table. ALTER TABLE some_table DROP IF EXISTS PARTITION(year = 2012); This command will remove the data and metadata for this partition.

Can we update partition column in Hive?

Hive ALTER TABLE command is used to update or drop a partition from a Hive Metastore and HDFS location (managed table). You can also manually update or drop a Hive partition directly on HDFS using Hadoop commands, if you do so you need to run the MSCK command to synch up HDFS files with Hive Metastore.

How do I refresh metadata in Hive?

To flush the metadata for all tables, use the INVALIDATE METADATA command. Because REFRESH table_name only works for tables that the current Impala node is already aware of, when you create a new table in the Hive shell, enter INVALIDATE METADATA new_table before you can see the new table in impala-shell.


2 Answers

You can update a Hive partition by, for example:

ALTER TABLE logs PARTITION(year = 2012, month = 12, day = 18)  SET LOCATION 'hdfs://user/darcy/logs/2012/12/18'; 

This command does not move the old data, nor does it delete the old data. It simply sets the partition to the new location.

To drop a partition, you can do

ALTER TABLE logs DROP IF EXISTS PARTITION(year = 2012, month = 12, day = 18); 

Hope it helps!

like image 79
darcyy Avatar answered Sep 18 '22 17:09

darcyy


in addition, you can drop multiple partitions from one statement (Dropping multiple partitions in Impala/Hive).

Extract from above link:

hive> alter table t drop if exists partition (p=1),partition (p=2),partition(p=3); Dropped the partition p=1 Dropped the partition p=2 Dropped the partition p=3 OK 

EDIT 1:

Also, you can drop bulk using a condition sign (>,<,<>), for example:

Alter table t  drop partition (PART_COL>1); 
like image 37
F.Lazarescu Avatar answered Sep 20 '22 17:09

F.Lazarescu