Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How do I setup ingress for Kibana in Kubernetes

I have just installed an EFK stack on my Kubernetes cluster using the guide on https://medium.com/@timfpark/efk-logging-on-kubernetes-on-azure-4c54402459c4

I have it working when accessing it through the proxy as stated in the guide on

http://localhost:8001/api/v1/namespaces/kube-system/services/kibana-logging/proxy

However, I want it to work through my existing ingress controller so I have created a new ingress rule using the yaml below:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  generation: 2
  labels:
    app: kibana
  name: kibana
  namespace: kube-system
spec:
  rules:
  - host: kibana.dev.example1.com
    http:
      paths:
      - backend:
          serviceName: kibana-logging
          servicePort: 5601
        path: /
status:
  loadBalancer:
    ingress:
    - {}

To my service which runs as:

apiVersion: v1
kind: Service
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    k8s-app: kibana-logging
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: Kibana
  name: kibana-logging
  namespace: kube-system
spec:
  clusterIP: X.X.195.49
  ports:
  - port: 5601
    protocol: TCP
    targetPort: ui
  selector:
    k8s-app: kibana-logging
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

However, when I try and access my URL: http://kibana.dev.example1.com

I get: {"statusCode":404,"error":"Not Found","message":"Not Found"}

If I try and access: http://kibana.dev.example1.com/app/kibana#

I get: "Kibana did not load properly. Check the server output for more information."

After looking through the logs for both Kibana pod and ingress pod and comparing the results between a successful request through the proxy and an unsuccessful request through the ingress I can see that...

for hitting /

"GET / HTTP/1.1" 200 197 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 491 0.003 [kube-system-kibana-logging-5601] X.X.22.204:5601 197 0.003 200 6101a7003003d34636d2012e53c23ca7
"GET /api/v1/namespaces/kube-system/services/kibana-logging/proxy/app/kibana HTTP/1.1" 404 85 "http://kibana.dev.example1.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 612 0.003 [kube-system-kibana-logging-5601] X.X.22.204:5601 85 0.003 404 5809ac2b33d3e23b200b13c9971d8520

for hitting /app/kibana#

"GET /app HTTP/1.1" 404 85 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 470 0.003 [kube-system-kibana-logging-5601] X.X.22.204:5601 85 0.003 404 54a4abe0cae6d3d4298847a0db0786d6
"GET /app/kibana HTTP/1.1" 200 13301 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 477 0.041 [kube-system-kibana-logging-5601] X.X.22.204:5601 13272 0.040 200 6cb7e7698f5c72e0cd06b3408d8d4673
"GET /api/v1/namespaces/kube-system/services/kibana-logging/proxy/bundles/kibana.style.css?v=16627 HTTP/1.1" 404 85 "https://kibana.dev.example1.com/app/kibana" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 576 0.004 [kube-system-kibana-logging-5601] X.X.22.204:5601 85 0.004 404 0b825f03c36c2225ab082c2a0bab15f4

When hitting through the proxy most of these requests return 302s rather than 404s. Can the ingress not access these URLs in namespace kube-system?

Am I missing something obvious here - maybe hitting the wrong Kibana URL? I've done a lot of googling and can't find anything similar.

like image 844
Garreth Avatar asked Oct 03 '18 13:10

Garreth


People also ask

How do you deploy Kibana in Kubernetes?

To launch Kibana on Kubernetes, we'll create a Service called kibana , and a Deployment consisting of one Pod replica. You can scale the number of replicas depending on your production needs, and optionally specify a LoadBalancer type for the Service to load balance requests across the Deployment pods.

What is the default ingress controller in Kubernetes?

The Kong Ingress Controller for Kubernetes is an ingress controller driving Kong Gateway. Kusk Gateway is an OpenAPI-driven ingress controller based on Envoy. The NGINX Ingress Controller for Kubernetes works with the NGINX webserver (as a proxy).

What object fulfills the ingress in Kubernetes?

An Ingress controller is responsible for fulfilling the Ingress, usually with a load balancer, though it may also configure your edge router or additional frontends to help handle the traffic.


2 Answers

Turns out the problem was with the kibana configuration.

In the kibana deployment yaml there is an environment variable called SERVER_BASEPATH which is set to point at the kibana service proxy. This was causing the URL to be rewritten each time I tried to access the endpoint externally.

If you comment out this variable and it's value and redeploy kibana then it should work by just hitting the ingress address.

e.g. http://kibana.dev.example1.com/

like image 105
Garreth Avatar answered Oct 27 '22 13:10

Garreth


turns out following environment variables needs to be set in order to expose kibana through ingress:

add following in env in kibana deployment.yaml:

        - name: ELASTICSEARCH_HOSTS
          value: "http://10.20.30.40:9200"
        - name: SERVER_BASEPATH
          value: "/kibana"
        - name: SERVER_REWRITEBASEPATH
          value: "true"
        - name: SERVER_PUBLICBASEURL
          value: "https://my.domain.com/kibana"

then use following to expose it over ingress:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  namespace: elastic
  name: gateway-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
    - host: my.domain.com
      http: 
        paths:
          - path: /kibana
            backend:
              serviceName: kibana
              servicePort: 5601
like image 44
Prerak Tiwari Avatar answered Oct 27 '22 12:10

Prerak Tiwari