Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to reduce number of output files in Apache Hive

Does anyone know of a tool that can "crunch" the output files of Apache Hadoop into fewer files or one file. Currently I am downloading all the files to a local machine and the concatenate them in one file. So does anyone know of an API or a tool that does the same. Thanks in advance.

like image 281
CruncherBigData Avatar asked May 12 '13 19:05

CruncherBigData


2 Answers

Limiting the number of output files means you want to limit the number of reducers. You could do that with the help of mapred.reduce.tasks property from the Hive shell. Example :

hive>  set mapred.reduce.tasks = 5;

But it might affect the performance of your query. Alternatively, you could use getmerge command from the HDFS shell once you are done with your query. This command takes a source directory and a destination file as input and concatenates files in src into the destination local file.

Usage :

bin/hadoop fs -getmerge <src> <localdst>

HTH

like image 159
Tariq Avatar answered Oct 25 '22 21:10

Tariq


See https://community.cloudera.com/t5/Support-Questions/Hive-Multiple-Small-Files/td-p/204038

set hive.merge.mapfiles=true;     -- Merge small files at the end of a map-only job.
set hive.merge.mapredfiles=true;  -- Merge small files at the end of a map-reduce job.

set hive.merge.size.per.task=???; -- Size (bytes) of merged files at the end of the job.

set hive.merge.smallfiles.avgsize=??? -- File size (bytes) threshold
-- When the average output file size of a job is less than this number, 
-- Hive will start an additional map-reduce job to merge the output files 
-- into bigger files. This is only done for map-only jobs if hive.merge.mapfiles 
-- is true, and for map-reduce jobs if hive.merge.mapredfiles is true.
like image 1
GreenGiant Avatar answered Oct 25 '22 21:10

GreenGiant