Kibana with nginx ingress controller in Kubernetes

Question

I am trying to get Kibana 6.2.4 in my GKE Kubernetes cluster running under www.mydomain.com/kibana without success. Though, I can run it perfectly fine with kubectl proxy and the default SERVER_BASEPATH.

Here is my Kibana deployment with the SERVER_BASEPATH removed.

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana-logging
  namespace: logging
  labels:
    k8s-app: kibana-logging
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: kibana-logging
  template:
    metadata:
      labels:
        k8s-app: kibana-logging
      annotations:
        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
      containers:
      - name: kibana-logging
        image: docker.elastic.co/kibana/kibana-oss:6.2.4
        resources:
          # need more cpu upon initialization, therefore burstable class
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        env:
          - name: ELASTICSEARCH_URL
            value: http://elasticsearch-logging:9200
          # - name: SERVER_BASEPATH
          #   value: /api/v1/namespaces/logging/services/kibana-logging/proxy
        ports:
        - containerPort: 5601
          name: ui
          protocol: TCP

My nginx ingress definition (nginx-ingress-controller:0.19.0):

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: logging-ingress
  namespace: logging
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/configuration-snippet: |
      rewrite ^/kibana/(.*)$ /$1 break;
spec:
  tls:
    - hosts:
      - dev.mydomain.net
      secretName: mydomain-net-tls-secret
  rules:
  - host: dev.mydomain.net
    http:
      paths:
      - path: /kibana
        backend:
          serviceName: kibana-logging
          servicePort: 5601

This results in this nginx location

    location /kibana {

        set $namespace      "logging";
        set $ingress_name   "logging-ingress";
        set $service_name   "kibana-logging";
        set $service_port   "5601";
        set $location_path  "/kibana";

        rewrite_by_lua_block {

            balancer.rewrite()

        }

        log_by_lua_block {

            balancer.log()

            monitor.call()
        }

        port_in_redirect off;

        set $proxy_upstream_name "logging-kibana-logging-5601";

        # enforce ssl on server side
        if ($redirect_to_https) {

            return 308 https://$best_http_host$request_uri;

        }

        client_max_body_size                    "1m";

        proxy_set_header Host                   $best_http_host;

        # Pass the extracted client certificate to the backend

        # Allow websocket connections
        proxy_set_header                        Upgrade           $http_upgrade;

        proxy_set_header                        Connection        $connection_upgrade;

        proxy_set_header X-Request-ID           $req_id;
        proxy_set_header X-Real-IP              $the_real_ip;

        proxy_set_header X-Forwarded-For        $the_real_ip;

        proxy_set_header X-Forwarded-Host       $best_http_host;
        proxy_set_header X-Forwarded-Port       $pass_port;
        proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

        proxy_set_header X-Original-URI         $request_uri;

        proxy_set_header X-Scheme               $pass_access_scheme;

        # Pass the original X-Forwarded-For
        proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

        # mitigate HTTPoxy Vulnerability
        # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
        proxy_set_header Proxy                  "";

        # Custom headers to proxied server

        proxy_connect_timeout                   5s;
        proxy_send_timeout                      60s;
        proxy_read_timeout                      60s;

        proxy_buffering                         "off";
        proxy_buffer_size                       "4k";
        proxy_buffers                           4 "4k";
        proxy_request_buffering                 "on";

        proxy_http_version                      1.1;

        proxy_cookie_domain                     off;
        proxy_cookie_path                       off;

        # In case of errors try the next upstream server before returning an error
        proxy_next_upstream                     error timeout;
        proxy_next_upstream_tries               3;

        rewrite ^/kibana/(.*)$ /$1 break;

        proxy_pass http://upstream_balancer;

        proxy_redirect                          off;

    }

However, going to /kibana results in a 404.

Stackdriver

2018-10-30 08:30:48.000 MDT
GET /kibana 404 61ms - 9.0B

Web page

{
  statusCode: 404,
  error: "Not Found",
  message: "Not Found"
}

I feel as though I am missing some sort of setting with either SERVER_BASEPATH and/or my nginx ingress configuration.

Answer 1

I believe what you want is the nginx.ingress.kubernetes.io/rewrite-target: / annotation in your ingress.

This way the location {} block will look something like this:

location ~* ^/kibana\/?(?<baseuri>.*) {
       ...
       rewrite (?i)/kibana/(.*) /$1 break;
       rewrite (?i)/kibana$ / break;
       ...
}