Hashicorp vault - Client sent an HTTP request to an HTTPS server - Readiness Probes

Question

Currently having a problem where the readiness probe is failing when deploying the Vault Helm chart. Vault is working but whenever I describe the pods get this error. How do I get the probe to use HTTPS instead of HTTP if anyone knows how to solve this I would be great as losing my mind slowly?

Kubectl Describe pod

Name:         vault-0
Namespace:    default
Priority:     0
Node:         ip-192-168-221-250.eu-west-2.compute.internal/192.168.221.250
Start Time:   Mon, 24 Aug 2020 16:41:59 +0100
Labels:       app.kubernetes.io/instance=vault
              app.kubernetes.io/name=vault
              component=server
              controller-revision-hash=vault-768cd675b9
              helm.sh/chart=vault-0.6.0
              statefulset.kubernetes.io/pod-name=vault-0
Annotations:  kubernetes.io/psp: eks.privileged
Status:       Running
IP:           192.168.221.251
IPs:
  IP:           192.168.221.251
Controlled By:  StatefulSet/vault
Containers:
  vault:
    Container ID:  docker://445d7cdc34cd01ef1d3a46f2d235cb20a94e48279db3fcdd84014d607af2fe1c
    Image:         vault:1.4.2
    Image ID:      docker-pullable://vault@sha256:12587718b79dc5aff542c410d0bcb97e7fa08a6b4a8d142c74464a9df0c76d4f
    Ports:         8200/TCP, 8201/TCP, 8202/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Command:
      /bin/sh
      -ec
    Args:
      sed -E "s/HOST_IP/${HOST_IP?}/g" /vault/config/extraconfig-from-values.hcl > /tmp/storageconfig.hcl;
      sed -Ei "s/POD_IP/${POD_IP?}/g" /tmp/storageconfig.hcl;
      /usr/local/bin/docker-entrypoint.sh vault server -config=/tmp/storageconfig.hcl

    State:          Running
      Started:      Mon, 24 Aug 2020 16:42:00 +0100
    Ready:          False
    Restart Count:  0
    Readiness:      exec [/bin/sh -ec vault status -tls-skip-verify] delay=5s timeout=5s period=3s #success=1 #failure=2
    Environment:
      HOST_IP:               (v1:status.hostIP)
      POD_IP:                (v1:status.podIP)
      VAULT_K8S_POD_NAME:   vault-0 (v1:metadata.name)
      VAULT_K8S_NAMESPACE:  default (v1:metadata.namespace)
      VAULT_ADDR:           http://127.0.0.1:8200
      VAULT_API_ADDR:       http://$(POD_IP):8200
      SKIP_CHOWN:           true
      SKIP_SETCAP:          true
      HOSTNAME:             vault-0 (v1:metadata.name)
      VAULT_CLUSTER_ADDR:   https://$(HOSTNAME).vault-internal:8201
      HOME:                 /home/vault
      VAULT_CACERT:         /vault/userconfig/vault-server-tls/vault.ca
    Mounts:
      /home/vault from home (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from vault-token-cv9vx (ro)
      /vault/config from config (rw)
      /vault/userconfig/vault-server-tls from userconfig-vault-server-tls (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      vault-config
    Optional:  false
  userconfig-vault-server-tls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  vault-server-tls
    Optional:    false
  home:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  vault-token-cv9vx:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  vault-token-cv9vx
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age   From                                                    Message
  ----     ------     ----  ----                                                    -------
  Normal   Scheduled  7s    default-scheduler                                       Successfully assigned default/vault-0 to ip-192-168-221-250.eu-west-2.compute.internal
  Normal   Pulled     6s    kubelet, ip-192-168-221-250.eu-west-2.compute.internal  Container image "vault:1.4.2" already present on machine
  Normal   Created    6s    kubelet, ip-192-168-221-250.eu-west-2.compute.internal  Created container vault
  Normal   Started    6s    kubelet, ip-192-168-221-250.eu-west-2.compute.internal  Started container vault
  Warning  Unhealthy  0s    kubelet, ip-192-168-221-250.eu-west-2.compute.internal  Readiness probe failed: Error checking seal status: Error making API request.

URL: GET http://127.0.0.1:8200/v1/sys/seal-status
Code: 400. Raw Message:

Client sent an HTTP request to an HTTPS server.

Vault Config File

# global:
#   tlsDisable: false

injector:
  enabled: false

server:
  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca

  extraVolumes:
  - type: secret
    name: vault-server-tls # Matches the ${SECRET_NAME} from above
  affinity: ""
  readinessProbe:
    enabled: true
    path: /v1/sys/health
  # # livelinessProbe:
  # #   enabled: true
  # #   path: /v1/sys/health?standbyok=true
  # #   initialDelaySeconds: 60
  ha:
    enabled: true

    config: |
      ui = true

      api_addr = "https://127.0.0.1:8200" # Unsure if this is correct

      storage "dynamodb" {
        ha_enabled = "true"
        region     = "eu-west-2"
        table      = "global-vault-data"
        access_key = "KEY"
        secret_key = "SECRET"
      }

      # listener "tcp" {
      #     address = "0.0.0.0:8200"
      #     tls_disable = "true"
      # }

      listener "tcp" {
        address = "0.0.0.0:8200"
        cluster_address = "0.0.0.0:8201"
        tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
        tls_key_file  = "/vault/userconfig/vault-server-tls/vault.key"
        tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca"
      }

      seal "awskms" {
        region     = "eu-west-2"
        access_key = "KEY"
        secret_key = "SECRET"
        kms_key_id = "ID"
      }
ui:
  enabled: true
  serviceType: LoadBalancer

Answer 1

In your environment variable definitions you have:

VAULT_ADDR:           http://127.0.0.1:8200

And non TLS is diable on your Vault configs (TLS enabled):

      listener "tcp" {
        address = "0.0.0.0:8200"
        cluster_address = "0.0.0.0:8201"
        tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
        tls_key_file  = "/vault/userconfig/vault-server-tls/vault.key"
        tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca"
      }

And your Readiness probe is executing in the pod:

vault status -tls-skip-verify

So that's trying to connect to http://127.0.0.1:8200, you can try changing the environment variable to use HTTPS: VAULT_ADDR=https://127.0.0.1:8200

You may have another (different) issue with your configs and env variable not matching:

K8s manifest:

VAULT_API_ADDR:       http://$(POD_IP):8200

Vault configs:

api_addr = "https://127.0.0.1:8200"

✌️

Answer 2

If you are on Mac add the Vault URL to your .zshrc or .bash_profile file. On the terminal open either .zshrc or .bash_profile file by doing this:

$ open .zshrc

Copy and paste this into it export VAULT_ADDR='http://127.0.0.1:8200'

Save the file by issuing on the terminal

$ source .zshrc