How can I drop all partitions currently loaded in a Hive table?
I can drop a single partition with alter table <table> drop partition(a=, b=...);
I can load all partitions with the recover partitions statement. But I cannot seem to drop all partitions.
I'm using the latest Hive version supported by EMR, 0.8.1.
Hive drop or delete partition is performed using ALTER TABLE tablename DROP command. Dropping a partition from a table removes the data from HDFS and from Hive Metastore.
You have to use pl/sql block for dropping partitions from a table with select query. Use listagg for making a comma separated list. Show activity on this post. You may use the following sql to generate DML for dropping multiple table partitions.
You cannot drop column directly from a table using command ALTER TABLE table_name drop col_name; The only way to drop column is using replace command.
As of version 0.9.0 you can use comparators in the drop partition statement which may be used to drop all partitions at once.
An example, taken from the drop_partitions_filter.q testcase :
create table ptestfilter (a string, b int) partitioned by (c string, d string);
alter table ptestfilter add partition (c='US', d=1);
alter table ptestfilter add partition (c='US', d=2);
alter table ptestFilter add partition (c='Uganda', d=2);
alter table ptestfilter add partition (c='Germany', d=2);
alter table ptestfilter add partition (c='Canada', d=3);
alter table ptestfilter add partition (c='Russia', d=3);
alter table ptestfilter add partition (c='Greece', d=2);
alter table ptestfilter add partition (c='India', d=3);
alter table ptestfilter add partition (c='France', d=4);
show partitions ptestfilter;
alter table ptestfilter drop partition (c>'0', d>'0');
show partitions ptestfilter;
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With