How do I setup ingress for Kibana in Kubernetes

Question

I have just installed an EFK stack on my Kubernetes cluster using the guide on https://medium.com/@timfpark/efk-logging-on-kubernetes-on-azure-4c54402459c4

I have it working when accessing it through the proxy as stated in the guide on

http://localhost:8001/api/v1/namespaces/kube-system/services/kibana-logging/proxy

However, I want it to work through my existing ingress controller so I have created a new ingress rule using the yaml below:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  generation: 2
  labels:
    app: kibana
  name: kibana
  namespace: kube-system
spec:
  rules:
  - host: kibana.dev.example1.com
    http:
      paths:
      - backend:
          serviceName: kibana-logging
          servicePort: 5601
        path: /
status:
  loadBalancer:
    ingress:
    - {}

To my service which runs as:

apiVersion: v1
kind: Service
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    k8s-app: kibana-logging
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: Kibana
  name: kibana-logging
  namespace: kube-system
spec:
  clusterIP: X.X.195.49
  ports:
  - port: 5601
    protocol: TCP
    targetPort: ui
  selector:
    k8s-app: kibana-logging
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

However, when I try and access my URL: http://kibana.dev.example1.com

I get: {"statusCode":404,"error":"Not Found","message":"Not Found"}

If I try and access: http://kibana.dev.example1.com/app/kibana#

I get: "Kibana did not load properly. Check the server output for more information."

After looking through the logs for both Kibana pod and ingress pod and comparing the results between a successful request through the proxy and an unsuccessful request through the ingress I can see that...

for hitting /

"GET / HTTP/1.1" 200 197 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 491 0.003 [kube-system-kibana-logging-5601] X.X.22.204:5601 197 0.003 200 6101a7003003d34636d2012e53c23ca7
"GET /api/v1/namespaces/kube-system/services/kibana-logging/proxy/app/kibana HTTP/1.1" 404 85 "http://kibana.dev.example1.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 612 0.003 [kube-system-kibana-logging-5601] X.X.22.204:5601 85 0.003 404 5809ac2b33d3e23b200b13c9971d8520

for hitting /app/kibana#

"GET /app HTTP/1.1" 404 85 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 470 0.003 [kube-system-kibana-logging-5601] X.X.22.204:5601 85 0.003 404 54a4abe0cae6d3d4298847a0db0786d6
"GET /app/kibana HTTP/1.1" 200 13301 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 477 0.041 [kube-system-kibana-logging-5601] X.X.22.204:5601 13272 0.040 200 6cb7e7698f5c72e0cd06b3408d8d4673
"GET /api/v1/namespaces/kube-system/services/kibana-logging/proxy/bundles/kibana.style.css?v=16627 HTTP/1.1" 404 85 "https://kibana.dev.example1.com/app/kibana" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 576 0.004 [kube-system-kibana-logging-5601] X.X.22.204:5601 85 0.004 404 0b825f03c36c2225ab082c2a0bab15f4

When hitting through the proxy most of these requests return 302s rather than 404s. Can the ingress not access these URLs in namespace kube-system?

Am I missing something obvious here - maybe hitting the wrong Kibana URL? I've done a lot of googling and can't find anything similar.

Answer 1

Turns out the problem was with the kibana configuration.

In the kibana deployment yaml there is an environment variable called SERVER_BASEPATH which is set to point at the kibana service proxy. This was causing the URL to be rewritten each time I tried to access the endpoint externally.

If you comment out this variable and it's value and redeploy kibana then it should work by just hitting the ingress address.

e.g. http://kibana.dev.example1.com/

Answer 2

turns out following environment variables needs to be set in order to expose kibana through ingress:

add following in env in kibana deployment.yaml:

        - name: ELASTICSEARCH_HOSTS
          value: "http://10.20.30.40:9200"
        - name: SERVER_BASEPATH
          value: "/kibana"
        - name: SERVER_REWRITEBASEPATH
          value: "true"
        - name: SERVER_PUBLICBASEURL
          value: "https://my.domain.com/kibana"

then use following to expose it over ingress:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  namespace: elastic
  name: gateway-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
    - host: my.domain.com
      http: 
        paths:
          - path: /kibana
            backend:
              serviceName: kibana
              servicePort: 5601