Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How does Hive decide when to use map reduce and when not to?

As a simple example,

select * from tablename;

DOES NOT kick in map reduce, while

select count(*) from tablename;

DOES. What is the general principle used to decide when to use map reduce (by hive)?

like image 954
Lazer Avatar asked Sep 19 '11 04:09

Lazer


3 Answers

In general, any sort of aggregation, such as min/max/count is going to require a MapReduce job. This isn't going to explain everything for you, probably.

Hive, in the style of many RDBMS, has an EXPLAIN keyword that will outline how your Hive query gets translated into MapReduce jobs. Try running explain on both your example queries and see what it is trying to do behind the scenes.

like image 149
Donald Miner Avatar answered Oct 18 '22 03:10

Donald Miner


Whenever we fire a query like select * from tablename, Hive reads the data file and fetches the entire data without doing any aggregation(min/max/count etc.). It'll call a FetchTask rather than a mapreduce task.

This is also an optimization technique in Hive. hive.fetch.task.conversion property can (i.e. FETCH task) minimize latency of map-reduce overhead.

This is like we are reading a hadoop file : hadoop fs -cat filename

But if we use select colNames from tablename, it requires a map-reduce job as it needs to extract the 'column' from each row by parsing it from the file it loads.

like image 32
Pardeep Sharma Avatar answered Oct 18 '22 05:10

Pardeep Sharma


select * from tablename;

Just reads raw data from files in HDFS, so it is much faster without MapReduce.

like image 1
wlk Avatar answered Oct 18 '22 04:10

wlk