Hadoop MRUnit throws exception

Question

I am trying to write some unit tests for my hadoop MR job and get following exception. This is the first time I am using MRUnit, so I am not really sure whats happening here.

java.lang.IncompatibleClassChangeError: Found class org.apache.hadoop.mapreduce.TaskInputOutputContext, but interface was expected
    at org.apache.hadoop.mrunit.mapreduce.mock.MockContextWrapper.createCommon(MockContextWrapper.java:53)
    at org.apache.hadoop.mrunit.mapreduce.mock.MockMapContextWrapper.create(MockMapContextWrapper.java:70)
    at org.apache.hadoop.mrunit.mapreduce.mock.MockMapContextWrapper.<init>(MockMapContextWrapper.java:62)
    at org.apache.hadoop.mrunit.mapreduce.MapDriver.run(MapDriver.java:217)
    at org.apache.hadoop.mrunit.MapDriverBase.runTest(MapDriverBase.java:150)
    at org.apache.hadoop.mrunit.TestDriver.runTest(TestDriver.java:137)

My actual code looks very simple

private MapDriver<Text, Text, Text, Text> mapDriver = MapDriver.newMapDriver(mapper);
private ReduceDriver<Text, Text, Text, Text> reduceDriver = ReduceDriver.newReduceDriver(reducer);
private MapReduceDriver<Text, Text, Text, Text, Text, Text> driver = MapReduceDriver.newMapReduceDriver(mapper, reducer);

@Test
public void testMapper() {
    mapDriver.withInput(new Text("1"), new Text("Line 1"));
    mapDriver.withOutput(new Text("1"), new Text("Line 1"));
    mapDriver.runTest();
}

Answer 1

As you can see in here, there are two classifiers in mrunit 0.9. And there are some problems using mrunit under Java 7 environment.

Perhaps, you are deveoping with hadoop 1.0.x and added only one of two classifiers in your pom.xml file. The solution is easy: Just add another one, like described below. That's all.

<dependencies>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-core</artifactId>
        <version>1.0.3</version>
    </dependency>
    <dependency>
        <groupId>org.apache.mrunit</groupId>
        <artifactId>mrunit</artifactId>
        <version>0.9.0-incubating</version>
        <classifier>hadoop1</classifier>
    </dependency>
    <dependency>
        <groupId>org.apache.mrunit</groupId>
        <artifactId>mrunit</artifactId>
        <version>0.9.0-incubating</version>
        <classifier>hadoop2</classifier>
    </dependency>
</dependencies>

I think this bug would be solved in next release, but I can not be convinced.

Answer 2

This error tends to happen if the version of MRUnit you are using was built against a different release of Hadoop than then code you are testing. In this particular case I believe TaskInputOutputContext was changed from an abstract class to an interface in 0.21.

So, it's probably the case that you are using 0.21 while the MRUnit version your using built against 0.20. I would simply work to get the versions sorted out.