Elasticsearch EC2 discovery fails with "invalid credentials"

Question

I'm trying to create an Elasticsearch cluster on EC2 and getting errors that I have supplied invalid credentials, but these are the exact same credentials I used when creating the instances via jclouds.

A sample error log I see from Elasticsearch:

[2014-03-03 21:32:26,109][INFO ][node                     ] [Baron Blood] version[1.0.1], pid[6832], build[5c03844/2014-02-25T15:52:53Z]
[2014-03-03 21:32:26,110][INFO ][node                     ] [Baron Blood] initializing ...
[2014-03-03 21:32:26,127][INFO ][plugins                  ] [Baron Blood] loaded [cloud-aws], sites []
[2014-03-03 21:32:30,736][INFO ][node                     ] [Baron Blood] initialized
[2014-03-03 21:32:30,736][INFO ][node                     ] [Baron Blood] starting ...
[2014-03-03 21:32:30,932][INFO ][transport                ] [Baron Blood] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.154.175.62:9300]}
[2014-03-03 21:32:31,228][WARN ][org.apache.http.impl.client.DefaultHttpClient] Authentication error: Unable to respond to any of these challenges: {}
[2014-03-03 21:32:31,388][INFO ][discovery.ec2            ] [Baron Blood] Exception while retrieving instance list from AWS API: AWS was not able to validate the provided access credentials
[2014-03-03 21:32:46,415][WARN ][org.apache.http.impl.client.DefaultHttpClient] Authentication error: Unable to respond to any of these challenges: {}
[2014-03-03 21:32:46,425][INFO ][discovery.ec2            ] [Baron Blood] Exception while retrieving instance list from AWS API: AWS was not able to validate the provided access credentials
[2014-03-03 21:33:00,939][WARN ][discovery                ] [Baron Blood] waited for 30s and no initial state was set by the discovery
[2014-03-03 21:33:00,939][INFO ][discovery                ] [Baron Blood] adstage-es-log/KolEM00zT9mYvYn3mDkrow
[2014-03-03 21:33:00,946][INFO ][http                     ] [Baron Blood] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.154.175.62:9200]}
[2014-03-03 21:33:00,998][INFO ][node                     ] [Baron Blood] started
[2014-03-03 21:33:01,454][WARN ][org.apache.http.impl.client.DefaultHttpClient] Authentication error: Unable to respond to any of these challenges: {}
[2014-03-03 21:33:01,463][INFO ][discovery.ec2            ] [Baron Blood] Exception while retrieving instance list from AWS API: AWS was not able to validate the provided access credentials
[2014-03-03 21:33:01,466][INFO ][cluster.service          ] [Baron Blood] new_master [Baron Blood][KolEM00zT9mYvYn3mDkrow][ip-10-154-175-62][inet[/10.154.175.62:9300]], reason: zen-disco-join (elected_as_master)
[2014-03-03 21:33:01,516][INFO ][gateway                  ] [Baron Blood] recovered [0] indices into cluster_state

My elasticsearch.yml file seems, based on the documentation, to be correct:

cluster.name: adstage-es-log
cloud.aws.access_key: MY_ACCESS_KEY
cloud.aws.secret_key: MY_SECRET_TOKEN
cloud.aws.region: us-east
discovery.type: ec2
discovery.ec2.ping_timeout: 30s

Also, because someone will ask, security group is wide open on these boxes. Using Elasticsearch 1.0.1 and ec2 plugin 2.0.0.RC1.

So far I haven't found anything indicating what could be causing this. Any ideas on how to address this issue?

Answer 1

This is it as a policy ready for copy&paste:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1404060922000",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeInstances",
        "ec2:DescribeRegions",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeTags"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

Answer 2

Turns out the issue was that I had incorrect permissions set for my AWS account since we're using IAM. Was able to get past this issue by making sure my account had the following permissions:

• ec2:DescribeAvailabilityZones
• ec2:DescribeInstances
• ec2:DescribeRegions
• ec2:DescribeSecurityGroups
• ec2:DescribeTags

Answer 3

To add a concrete example to Gordon's helpful answer, here is a working IAM permissions policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ec2:Describe*",
      "Resource": "*"
    }
  ]
}