How to connect to AWS Elasticsearch using the Elasticsearch JavaScript SDK?

Question

I'm using the AWS Elasticsearch service and would like to connect via elasticsearch.js, but a port is required.

It looks like AWS only offers the REST API (e.g. via curl), running on port 80. My cluster is up and I can access via the browser, but not via elasticsearch.js.

This example doesn't work for me:

var elasticsearch = require('elasticsearch');
var client = new elasticsearch.Client({
  host: 'localhost:9200', // not working: '', 80, default: 443
  log: 'trace'
});

client.ping({
  requestTimeout: 1000
}, function (error) {
  if (error) {
    console.trace('elasticsearch cluster is down!');
  } else {
    console.log('All is well');
  }
});

I found http-aws-es, but it's not working either.

Any idea? Thanks in advance!

Answer 1

For the elasticsearch.client, you can use http-aws-es for connectionClass and amazonES with keys.

var client = new elasticsearch.Client({
    hosts: config.elasticsearch.host,
    connectionClass: require('http-aws-es'),
    amazonES: {
        region: config.aws.region,
        accessKey: config.aws.key,
        secretKey: config.aws.secret
    }
});

Answer 2

NPM package elasticsearch has been deprecated, and replaced by @elastic/elasticsearch

So instead of using http-aws-es, which is supposed to work with the deprecated elasticsearch package, you can consider using package @acuris/aws-es-connection, an AWS ES connection for the new elasticsearch client @elastic/elasticsearch. It works well for me in my project. You can find its usage in its readme file, but here is a piece of sample code:

import {
  createAWSConnection,
  awsGetCredentials,
} from '@acuris/aws-es-connection';
import { Client } from '@elastic/elasticsearch';

export const getESClient = async () => {
  const esEndpoint = process.env.AWS_ES_ENDPOINT;
  if (!esEndpoint) {
    throw new Error(
      'AWS_ES_ENDPOINT ENV not set.'
    );
  }

  const awsCredentials = await awsGetCredentials();
  const AWSConnection = createAWSConnection(awsCredentials);
  const client = new Client({
    ...AWSConnection,
    node: esEndpoint,
  });
  return client;
};

export const createNewIndex = async (index: string) => {
  try {
    const client = await getESClient();
    const exists = await client.indices.exists({ index });
    if (!exists || !exists.statusCode || exists.statusCode !== 404) {
      console.log(`Index ${index} might alrady exist.`, exists);
      return false;
    }
    const created = await client.indices.create({
      index,
      body: {
        mappings: {
          properties: {
            product_id: {
              type: 'keyword',
            },
            product_description: {
              type: 'text',
            },
          },
        },
      },
    });
    console.log(`Index created for ${index}`, created);
  } catch (error) {
    console.log(`Error creating index ${index}`, error);
    return false;
  }
  return true;
};

Answer 3

Here's an implementation of the Connection class I've been using with TypeScript:

import { Connection as UnsignedConnection } from '@elastic/elasticsearch';
import * as AWS from 'aws-sdk';
import RequestSigner from 'aws-sdk/lib/signers/v4';
import { ClientRequest, IncomingMessage } from 'http';

class AwsElasticsearchError extends Error {}
type RequestOptions = Parameters<UnsignedConnection['request']>[0];


class AwsSignedConnection extends UnsignedConnection {
  public request(
    params: RequestOptions,
    callback: (err: Error | null, response: IncomingMessage | null) => void,
  ): ClientRequest {
    const signedParams = this.signParams(params);
    return super.request(signedParams, callback);
  }

  private signParams(params: RequestOptions): RequestOptions {
    const region = AWS.config.region || process.env.AWS_DEFAULT_REGION;
    if (!region) throw new AwsElasticsearchError('missing region configuration');
    if (!params.method) throw new AwsElasticsearchError('missing request method');
    if (!params.path) throw new AwsElasticsearchError('missing request path');
    if (!params.headers) throw new AwsElasticsearchError('missing request headers');

    const endpoint = new AWS.Endpoint(this.url.href);
    const request = new AWS.HttpRequest(endpoint, region);

    request.method = params.method;
    request.path = params.querystring
      ? `${params.path}/?${params.querystring}`
      : params.path;
    request.body = params.body;

    Object.entries(params.headers).forEach(([header, value]) => {
      if (value === undefined) return;
      if (typeof value === 'string') request.headers[header] = value;
      else if (typeof value === 'number') request.headers[header] = `${value}`;
      else request.headers[header] = value.join('; ');
    });
    request.headers.Host = endpoint.host;

    const signer = new RequestSigner(request, 'es');
    signer.addAuthorization(AWS.config.credentials, new Date());
    return request;
  }
}

export { AwsSignedConnection, UnsignedConnection, AwsElasticsearchError };

Then you can provide it only if credentials are available, so you can use it to point to a local (e.g. Docker) Elasticsearch without credentials:

import awsSdk from 'aws-sdk';
import elasticsearch from '@elastic/elasticsearch';
import { AwsSignedConnection, UnsignedConnection } from '../aws-es-connector';

client = new elasticsearch.Client({
  Connection: awsSdk.config.credentials ? AwsSignedConnection : UnsignedConnection,
  node: elasticsearchEndpoint,
});