Searching against secured AWS ElasticSearch

Question

I have setup a new ElasticSearch cluster on AWS which is only allowing access to a specific IAM user.

However, I'm trying to connect to this from Ruby and looked at using the AWS SDK but that has no methods for actually making HTTP operations against your ES cluster, only accessing the configuration APIs.

As usual, this requires all the AWS request signing stuff that they require for API access, but I can't find anything that indicates how to do this stuff. I'm using Ruby.

Essentially, what I'm after is being able to make GET and PUT requests to this cluster using the IAM user creds. IP restriction isn't an option for me.

Answer 1

You can make signed, secure requests to Amazon Elasticsearch from Ruby. I did the following with an app on Heroku.

Ensure you have elasticsearch gem >= v1.0.15 as support for this was only implemented there Dec 4th, 2015.

You also need this gem:

gem 'faraday_middleware-aws-signers-v4'

Example from the elasticsearch-ruby/elasticsearch-transport documentation:

You can use any standard Faraday middleware and plugins in the configuration block, for example sign the requests for the AWS Elasticsearch service:

With the following code:

require 'faraday_middleware/aws_signers_v4'

client = Elasticsearch::Client.new(url: ENV['AWS_ENDPOINT_URL']) do |f|
  f.request :aws_signers_v4,
            credentials: Aws::Credentials.new(ENV['AWS_ACCESS_KEY_ID'], ENV['AWS_SECRET_ACCESS_KEY']),
            service_name: 'es',
            region: 'us-east-1'
  f.adapter Faraday.default_adapter
end

This also works with the searchkick gem with Rails. Set Searchkick.client using the above example, in an initializer:

# config/initializers/elasticsearch.rb
require 'faraday_middleware/aws_signers_v4'

Searchkick.client = Elasticsearch::Client.new(url: ENV['AWS_ENDPOINT_URL']) do |f|
  f.request :aws_signers_v4,
            credentials: Aws::Credentials.new(ENV['AWS_ACCESS_KEY_ID'], ENV['AWS_SECRET_ACCESS_KEY']),
            service_name: 'es',
            region: 'us-east-1'
  f.adapter Faraday.default_adapter
end