Deployment of servless app fails - Enable fine-grained access control or apply a restrictive access policy to your domain

Question

I have a serverless app where I would like to deploy an elasticsearch cluster. I have configured it like this:

PostsSearch:
      Type: AWS::Elasticsearch::Domain
      Properties:
        ElasticsearchVersion: '6.3'
        DomainName: images-search-${self:provider.stage}
        ElasticsearchClusterConfig:
          DedicatedMasterEnabled: false
          InstanceCount: 1
          ZoneAwarenessEnabled: false
          InstanceType: t2.small.elasticsearch
        EBSOptions:
          EBSEnabled: true
          Iops: 0
          VolumeSize: 10
          VolumeType: 'gp2'
        AccessPolicies:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Principal:
                AWS: '*'
              Action: 'es:ESHttp*'
              Resource: '*'

But, I get an error:

An error occurred: PostsSearch - Enable fine-grained access control or apply a restrictive access policy to your domain (Service: AWSElasticsearch; Status Code: 400; Error Code: ValidationException; Request ID: be0eca95-23ae-4ac9-be81-67cab37ccd70; Proxy: null).

How should I fix this?

Answer 1

Based on the extra discussion in the comments.

It is not possible to make an ES domain totally public. CloudFormation will not allow for that. Thus, there are three options to choose from. Below I present three of them with in a sample serverless application. This is just basic hello-world application, it does not use the ES domain in any capacity, but I use it to verify that each choice works and can be deployed using serverless framework without errors.

Apply IP-based condition

This will make your domain open for access to only individual IP address or IP CIDR range. The example below limits access to one, single IP address.

service: estest

provider:
  name: aws
  runtime: python3.8

  stage: ${opt:stage, 'dev'}
  region: ${opt:region, 'us-east-1'}  

functions:
  hello:
    handler: handler.hello

resources:
  Resources:
    PostsSearch:
      Type: AWS::Elasticsearch::Domain
      Properties:
        ElasticsearchVersion: '6.3'
        DomainName: images-search-${self:provider.stage}
        ElasticsearchClusterConfig:
          DedicatedMasterEnabled: false
          InstanceCount: 1
          ZoneAwarenessEnabled: false
          InstanceType: t2.small.elasticsearch
        EBSOptions:
          EBSEnabled: true
          Iops: 0
          VolumeSize: 10
          VolumeType: 'gp2'
        AccessPolicies:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Principal:
                AWS: '*'
              Action: 'es:ESHttp*'
              Resource: !Sub "arn:aws:es:${self:provider.region}:${AWS::AccountId}:domain/images-search-${self:provider.stage}/*"
              Condition:
                IpAddress:
                   aws:SourceIp: ["12.13.14.15"] 

Restrict principal

You can restrict access to your ES domain to selected IAM user or role. This way, only the given IAM user/role will be able to access the ES domain. In the below I use lambda existing IAM role as a principle. The function and its role must already exist.

service: estest

provider:
  name: aws
  runtime: python3.8

  stage: ${opt:stage, 'dev'}
  region: ${opt:region, 'us-east-1'}  

functions:
  hello:
    handler: handler.hello

resources:
  Resources:
    PostsSearch:
      Type: AWS::Elasticsearch::Domain
      Properties:
        ElasticsearchVersion: '6.3'
        DomainName: images-search-${self:provider.stage}
        ElasticsearchClusterConfig:
          DedicatedMasterEnabled: false
          InstanceCount: 1
          ZoneAwarenessEnabled: false
          InstanceType: t2.small.elasticsearch
        EBSOptions:
          EBSEnabled: true
          Iops: 0
          VolumeSize: 10
          VolumeType: 'gp2'
        AccessPolicies:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Action: 'es:ESHttp*'            
              Principal:
                AWS: !Sub "arn:aws:iam::${AWS::AccountId}:role/service-role/lambda-function-es-role-b44mvudf"
              Resource: !Sub "arn:aws:es:${self:provider.region}:${AWS::AccountId}:domain/images-search-${self:provider.stage}/*"

Use fine-grained access control

The example here creates publicly accessible ES domain with fine-grained controls that requires username and password. This does not work in free-tier. I also hard-coded username and password, which obviously would need to be modified and provided as a parameter from from SSM Parameter store in real application.

service: estest

provider:
  name: aws
  runtime: python3.8

  stage: ${opt:stage, 'dev'}
  region: ${opt:region, 'us-east-1'}  

functions:
  hello:
    handler: handler.hello

resources:
  Resources:
    PostsSearch:    
      Type: AWS::Elasticsearch::Domain
      Properties: 
        DomainName: images-search-${self:provider.stage}        
        AccessPolicies: !Sub |
          {
            "Version": "2012-10-17",
            "Statement": [
              {
                "Effect": "Allow",
                "Principal": {
                  "AWS": "*"
                },
                "Action": "es:*",
                "Resource": "*"
              }
            ]
          }
        AdvancedSecurityOptions:
            Enabled: true
            InternalUserDatabaseEnabled: true
            MasterUserOptions: 
              MasterUserName: admin
              MasterUserPassword: fD343sfdf!3rf
        EncryptionAtRestOptions: 
          Enabled: true
        NodeToNodeEncryptionOptions:
          Enabled: true
        DomainEndpointOptions:
          EnforceHTTPS: true
        EBSOptions: 
          EBSEnabled: true
          VolumeSize: 20
          VolumeType: gp2
        ElasticsearchClusterConfig: 
          DedicatedMasterEnabled: false
          InstanceCount: 1
          InstanceType: c4.large.elasticsearch
          ZoneAwarenessEnabled: false
        ElasticsearchVersion: 7.7