Can't connect to Elasticsearch with Node.Js on Kubernetes (self signed certificate in certificate chain)

Question

I have a NodeJs application running inside a Kubernetes cluster (I am using microk8s). I've also followed the official steps to setup Elasticsearch on Kubernetes.

Issue

But I am unable to connect to the Elasticsearch cluster. I get this error:

ConnectionError: self signed certificate in certificate chain

This is a code snippet of my connection:

const client = new elasticsearch.Client({
  node: process.env.elasticsearch_node,
  // https://elasticsearch-es-http.default.svc.cluster.local:9200
});

Minimal reproduction

I've created a minimal reproduction of this issue here: https://github.com/flolu/elasticsearch-k8s-connection. (Setup instructions are in the README)

Basically, everything works fine when running Elasticsearch inside Docker compose, but I can't connect when running inside Kubernetes.

The reason for this is probably because I didn't setup TLS certificates correctly, but I haven't found any information about it. Do I configure it inside my NodeJs application when creating the ES client or at a cluster level?

Answer 1

The solution is to configure SSL and the Elastic user when creating the Client

const client = new elasticsearch.Client({
  node: process.env.elasticsearch_node,
  auth: {
    username: "elastic",
    password: process.env.elasticsearch_password || "changeme",
  },
  ssl: {
    ca: process.env.elasticsearch_certificate,
    rejectUnauthorized: false,
  },
});

The password and the certificate are provided by Elastic. They are stored inside Kubernetes secrets. So I've just passed the password and the certificate into my NodeJs service via environment variables like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: search-deployment
spec:
  selector:
    matchLabels:
      app: search
  replicas: 1
  template:
    metadata:
      labels:
        app: search
    spec:
      containers:
        - name: search
          image: search:placeholder_name
          imagePullPolicy: Always
          env:
            - name: elasticsearch_node
              value: https://elasticsearch-es-http.default.svc.cluster.local:9200
            - name: elasticsearch_certificate
              valueFrom:
                secretKeyRef:
                  name: elasticsearch-es-http-ca-internal
                  key: tls.crt
            - name: elasticsearch_password
              valueFrom:
                secretKeyRef:
                  name: elasticsearch-es-elastic-user
                  key: elastic

Answer 2

This is an SSL problem Please try to disable verifying SSL or apply for an SSL from CA. Cloudflare SSL is also a good choice .

Here's how to disable verifying SSL for NodeJs

process.env.NODE_TLS_REJECT_UNAUTHORIZED = '0';

Answer 3

I'd like to build on top of @Florian Ludewig's answer on 2 points, since I struggled myself to have it work on my side.

1. Don't turn off rejectUnauthorized

const client = new elasticsearch.Client({
  node: 'node httpS url here',
  ssl: {
    ca: process.env.elasticsearch_certificate,
    rejectUnauthorized: true, // <-- this is important
  },
});

If you set rejectUnauthorized to false, the underlying nodejs https agent will bypass the certificate check. Of course if you are confident in the security of your cluster, you could disable it, but it renders the idea of providing the CA cert useless in the first place.

2. Make sure you provide a PEM CA cert - not the base64 encoded version

Perhaps you are providing the CA cert from your own config file without using Kubernetes' secret injection - possibly because the ES client application is in a different namespace and thus cannot access the CA secret.

In this case you may find it useful to store the CA cert as a base64 string, in a config file, but you should not forget to provide a decoded string to your client:

const config = loadConfigFromFile('config.yml');
const caCertificate = Buffer.from(config.base64CaCertificate, 'base64').toString();

const client = new elasticsearch.Client({
  node: 'node httpS url here',
  ssl: {
    ca: caCertificate,
    rejectUnauthorized: true
  },
});