Maven dependencies for Hadoop: MiniDFSCluster & MiniMRCluster

Question

I want to implement a maven project, that helps me unit test a Hadoop MapReduce job. My biggest problem is defining the Maven dependencies to be able to make use of the test classes: MiniDFSCluster & MiniMRCluster.

I am using Hadoop 2.4.1. Any ideas?

Answer 1

In case someone else is still searchinf for an answer:

MiniMRCluster is now deprecated.

You can get MiniDFSCluster and MiniMRCluster in the dependency (shown for Gradle)

compile group: 'org.apache.hadoop', name: 'hadoop-minicluster', version: '2.7.2'

The dependency is basically just a pom file that lists out the dependencies in this package. For those who want to look this up, MiniDFSCluster is in the artifact hadoop-hdfs:tests

You don't have to use the dependencies from the Cloudera repository