InstanceProfile is required for creating cluster

Question

I was trying to run Elastic MapReduce from Eclipse but couldn't do so.

My code is as below:

public class RunEMR {

    /**
     * @param args
     */
    public static void main(String[] args) {
        // TODO Auto-generated method stub
         AWSCredentials credentials = new BasicAWSCredentials("xxxx","xxxx");
            AmazonElasticMapReduceClient emr = new AmazonElasticMapReduceClient(credentials);

            StepFactory stepFactory = new StepFactory();

            StepConfig enableDebugging = new StepConfig()
                .withName("Enable Debugging")
                .withActionOnFailure("TERMINATE_JOB_FLOW")
                .withHadoopJarStep(stepFactory.newEnableDebuggingStep());

            StepConfig installHive = new StepConfig()
                .withName("Install Hive")
                .withActionOnFailure("TERMINATE_JOB_FLOW")
                .withHadoopJarStep(stepFactory.newInstallHiveStep());

            StepConfig hiveScript = new StepConfig().withName("Hive Script")
                .withActionOnFailure("TERMINATE_JOB_FLOW")
                .withHadoopJarStep(stepFactory.newRunHiveScriptStep("s3://mywordcountbuckett/binary/WordCount.jar"));

            RunJobFlowRequest request = new RunJobFlowRequest()
                .withName("Hive Interactive")
                .withSteps(enableDebugging, installHive)
                .withLogUri("s3://mywordcountbuckett/")
                .withInstances(new JobFlowInstancesConfig()
                    .withEc2KeyName("xxxx")
                    .withHadoopVersion("0.20")
                    .withInstanceCount(3)
                    .withKeepJobFlowAliveWhenNoSteps(true)
                    .withMasterInstanceType("m1.small")
                    .withSlaveInstanceType("m1.small"));

            RunJobFlowResult result = emr.runJobFlow(request);



    }

}

The error that I got was :

Exception in thread "main" com.amazonaws.AmazonServiceException: InstanceProfile is required for creating cluster. (Service: AmazonElasticMapReduce; Status Code: 400; Error Code: ValidationException; Request ID: 7a96ee32-9744-11e5-947d-65ca8f7db0a5

I have tried for couple of hours but unable to fix it. Does anyone knows how ?

Answer 1

I got same exception InstanceProfile is required for creating cluster.

Had to set service-role, and job-flow-role like below

aRunJobFlowRequest.setServiceRole("EMR_DefaultRole")
aRunJobFlowRequest.setJobFlowRole("EMR_EC2_DefaultRole")

After that I was OK.

---

AWS Document for EMR IAM Roles said

AWS Identity and Access Management (IAM) roles provide a way for IAM users or AWS services to have certain specified permissions and access to resources. For example, this may allow users to access resources or other services to act on your behalf. You must specify two IAM roles for a cluster: a role for the Amazon EMR service (service role), and a role for the EC2 instances (instance profile) that Amazon EMR manages.

So word InstanceProfile in exception message might mean a role for the EC2 instances (instance profile) in the doc, but I got pass that exception after specifying JobFlowRole. little weird.

Answer 2

For an ec2 role (here jobflowrole), an instance profile with the same nameis created internally. Hence it uses these names interchangeably. If you creating an emr cluster from scratch using boto3, you should also create emr service role, one ec2jobflow role , one instance profile linked to ec2jobflow role. AWS doc