Hive unable to manually set number of reducers

Question

I have the following hive query:

select count(distinct id) as total from mytable; 

which automatically spawns:
1408 Mappers
1 Reducer

I need to manually set the number of reducers and I have tried the following:

set mapred.reduce.tasks=50  set hive.exec.reducers.max=50 

but none of these settings seem to be honored. The query takes forever to run. Is there a way to manually set the reducers or maybe rewrite the query so it can result in more reducers? Thanks!

Answer 1

writing query in hive like this:

 SELECT COUNT(DISTINCT id) .... 

will always result in using only one reducer. You should:

1. use this command to set desired number of reducers:

   set mapred.reduce.tasks=50

2. rewrite query as following:

SELECT COUNT(*) FROM ( SELECT DISTINCT id FROM ... ) t;

This will result in 2 map+reduce jobs instead of one, but performance gain will be substantial.

Answer 2

Number of reducers depends also on size of the input file

By default it is 1GB (1000000000 bytes). You could change that by setting the property hive.exec.reducers.bytes.per.reducer:

1. either by changing hive-site.xml

   <property>    <name>hive.exec.reducers.bytes.per.reducer</name>    <value>1000000</value> </property> 

2. or using set

   $ hive -e "set hive.exec.reducers.bytes.per.reducer=1000000"