Cannot Launch EMR Jobflow using Temporary Credentials from EC2 IAM Roles

Question

I have an instance which has been assigned an EC2 IAM role. I cannot create an EMR jobflow from this instance using the temporary credentials that having a Role assigned instance provides, I get the following response from the API:

  <Error>
    <Type>Sender</Type>
    <Code>ValidationError</Code>
    <Message>Service role and InstanceProfile are required for calls made with temporary credentials provided by STS</Message>
  </Error>

Googling this error message has revealed absolutely nothing. I get the same response from the API whether I use the AWS CLI or boto directly. In an attempt to follow the advice of this error message, I have tried passing {InstanceProfile: <instance_profile_name>} to the api_params argument for the run_jobflow method in boto, but still get the same error. I also tried using the service_role argument for run_jobflow, that also failed. Passing both together also failed.

According to this page from Amazon docs, EMR should support STS and EC2 IAM roles, so wondering if anyone has gotten this to work before.

Answer 1

I got this working in java. Like Sam, I specified a service role and jobflow role. It was a surprise to me that an error about InstanceProfile required that I set jobflow role. ex:

myRunJobFlowRequest.setServiceRole("EMR_DefaultRole");
myRunJobFlowRequest.setJobFlowRole("EMR_EC2_DefaultRole")

The other thing I did was in my cloud formation template, I granted the AmazonElasticMapReduceFullAccess policy to my instance role. ex:

"Policies": [
          {
            "PolicyName": "AmazonElasticMapReduceFullAccess",
            "PolicyDocument": {
              "Statement": [
                {
                  "Effect": "Allow",
                  "Action": "*",
                  "Resource": "*"
                }
              ]
            }
          }

Answer 2

Yes, I managed to get this working, using:

• An explicit service-role (using the EMR_DefaultRole as a template)
• An explicit jobflow-role (using the EMR_EC2_DefaultRole as a template)
• Granting iam:PassRole permissions (for the service-role resource) to the EC2 IAM role