Role issue using AWS ElasticSearch with S3

Question

I have an ElasticSearch instance up in the AWS cloud, which works fine. But I need to set it up with S3 to store snapshots.

First I need to setup the S3 repository, by posting the following to the endpoint (mock json example):

PUT _snapshot/my_s3_repository
{
   "type": "s3",
   "settings": {
      "bucket": "my_bucket_name",
      "region": "us-west"
    }
}

But in return i get the following:
"Message": "settings.role_arn is needed for snapshot registration."

Any ideas? I have tried messing around with roles in IAM with no luck.

Answer 1

Copying over what is from the AWS Forums thread to here in case the forum disappears:

---

The registration process for snapshots S3 repositories requires a role and a signed request.

To create the signed request to register the S3 endpoint you can use a python script. See below example.

All this process is described here: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains.html But, to summarize, you can follow these steps:

1) Create an IAM policy and add attach it to a role:

An example Role looks like the following:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "es.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
} 

An example policy like this one should be attached to previous role:

{
    "Version":"2012-10-17",
    "Statement":[
        {
            "Action":[
                "s3:ListBucket"
            ],
            "Effect":"Allow",
            "Resource":[
                "arn:aws:s3:::es-index-backups"
            ]
        },
        {
            "Action":[
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "iam:PassRole"
            ],
            "Effect":"Allow",
            "Resource":[
                "arn:aws:s3:::es-index-backups/*"
            ]
        }
    ]
} 

2) Registering a Snapshot Directory

As an IAM user with access to the new role, you must register the snapshot directory with Amazon Elasticsearch Service before you take manual index snapshots. This one-time operation requires that you sign your AWS request with the IAM role that grants permissions to Amazon ES.

Save the following sample Python code and modify the following values: region: The AWS region where you created the snapshot repository endpoint: The endpoint for your Amazon ES domain aws_access_key_id: IAM credential aws_secret_access_key: IAM credential path: The location of the snapshot repository

Note: The Python client requires the boto package be installed on the computer where you will register your snapshot repository. from boto.connection import AWSAuthConnection

class ESConnection(AWSAuthConnection):

    def __init__(self, region, **kwargs):
        super(ESConnection, self).__init__(**kwargs)
        self._set_auth_region_name(region)
        self._set_auth_service_name("es")

    def _required_auth_capability(self):
        return ['hmac-v4']

if __name__ == "__main__":

    client = ESConnection(
            region='us-east-1',
            host='search-weblogs-etrt4mbbu254nsfupy6oiytuz4.us-east-1.es.a9.com',
            aws_access_key_id='my-access-key-id',
            aws_secret_access_key='my-access-key', is_secure=False)

    print 'Registering Snapshot Repository'
    resp = client.make_request(method='PUT',
            path='/_snapshot/weblogs-index-backups',
            data='{"type": "s3","settings": { "bucket": "es-index-backups","region": "us-east-1","role_arn": "arn:aws:iam::123456789012:role/MyElasticsearchRole"}}')
    body = resp.read()
    print body

Once the S3 repository is registered, you will be able to manually take and restore snapshots using curl. As example:

To manually take a snapshot:

curl -XPUT 'http://<Elasticsearch_domain_endpoint>/_snapshot/snapshot_repository/snapshot_name'

To manually restore a snapshot:

curl -XPOST 'http://search-weblogs-abcdefghijklmnojiu.us-east-1.a9.com/_snapshot/weblogs-index-backups/snapshot_1/_restore'

Note: You cannot restore a snapshot of your indices to an Amazon ES cluster that already contains indices with the same names. Currently, Amazon ES does not support the Elasticsearch _close API, so you must use one of the following alternatives: Delete the indices on the same Amazon ES domain, then restore the snapshot Restore the snapshot to a different Amazon ES domain