Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

how to access Mapper Counter value in a Reducer?

I want to acces the myCounter.my value in reducer :

public static class Map extends Mapper<LongWritable, Text, ImmutableBytesWritable, ImmutableBytesWritable>
{
    public static enum myCounter{my};

    @Override
    public void map(LongWritable key, Text value, Context context) 
    {
        context.getCounter(myCounter.my).increment(1);
        context.write( new ImmutableBytesWritable ( ),new ImmutableBytesWritable() );
    }
}


public static class Reduce extends Reducer<ImmutableBytesWritable, ImmutableBytesWritable, Text, Text>
{
    @Override
    public void reduce(ImmutableBytesWritable key,Iterable<ImmutableBytesWritable> result,Context context)
    {

    }
}

Accessing a mapper's counter from a reducer(for old API is given ) how to make it work for new API ?

Or

I want to know the total number of mapper output ? Is there any better way ? (i am not able to access counter in Reducer:

Group Name->org.apache.hadoop.mapred.Task$Counter Counter Name->MAP_OUTPUT_RECORDS)

Thanks

like image 367
saurabh shashank Avatar asked Sep 25 '12 06:09

saurabh shashank


1 Answers

You to make it work for new API by accessing the counters via job object.

Configuration conf = context.getConfiguration();
Cluster cluster = new Cluster(conf);
Job currentJob = cluster.getJob(context.getJobID());
long val=currentJob.getCounters().findCounter(myCounter.my).getValue();
like image 89
Yuva Raj Avatar answered Oct 20 '22 15:10

Yuva Raj