I have just installed an EFK stack on my Kubernetes cluster using the guide on https://medium.com/@timfpark/efk-logging-on-kubernetes-on-azure-4c54402459c4
I have it working when accessing it through the proxy as stated in the guide on
http://localhost:8001/api/v1/namespaces/kube-system/services/kibana-logging/proxy
However, I want it to work through my existing ingress controller so I have created a new ingress rule using the yaml below:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
generation: 2
labels:
app: kibana
name: kibana
namespace: kube-system
spec:
rules:
- host: kibana.dev.example1.com
http:
paths:
- backend:
serviceName: kibana-logging
servicePort: 5601
path: /
status:
loadBalancer:
ingress:
- {}
To my service which runs as:
apiVersion: v1
kind: Service
metadata:
labels:
addonmanager.kubernetes.io/mode: Reconcile
k8s-app: kibana-logging
kubernetes.io/cluster-service: "true"
kubernetes.io/name: Kibana
name: kibana-logging
namespace: kube-system
spec:
clusterIP: X.X.195.49
ports:
- port: 5601
protocol: TCP
targetPort: ui
selector:
k8s-app: kibana-logging
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
However, when I try and access my URL:
http://kibana.dev.example1.com
I get: {"statusCode":404,"error":"Not Found","message":"Not Found"}
If I try and access: http://kibana.dev.example1.com/app/kibana#
I get: "Kibana did not load properly. Check the server output for more information."
After looking through the logs for both Kibana pod and ingress pod and comparing the results between a successful request through the proxy and an unsuccessful request through the ingress I can see that...
for hitting /
"GET / HTTP/1.1" 200 197 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 491 0.003 [kube-system-kibana-logging-5601] X.X.22.204:5601 197 0.003 200 6101a7003003d34636d2012e53c23ca7
"GET /api/v1/namespaces/kube-system/services/kibana-logging/proxy/app/kibana HTTP/1.1" 404 85 "http://kibana.dev.example1.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 612 0.003 [kube-system-kibana-logging-5601] X.X.22.204:5601 85 0.003 404 5809ac2b33d3e23b200b13c9971d8520
for hitting /app/kibana#
"GET /app HTTP/1.1" 404 85 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 470 0.003 [kube-system-kibana-logging-5601] X.X.22.204:5601 85 0.003 404 54a4abe0cae6d3d4298847a0db0786d6
"GET /app/kibana HTTP/1.1" 200 13301 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 477 0.041 [kube-system-kibana-logging-5601] X.X.22.204:5601 13272 0.040 200 6cb7e7698f5c72e0cd06b3408d8d4673
"GET /api/v1/namespaces/kube-system/services/kibana-logging/proxy/bundles/kibana.style.css?v=16627 HTTP/1.1" 404 85 "https://kibana.dev.example1.com/app/kibana" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 576 0.004 [kube-system-kibana-logging-5601] X.X.22.204:5601 85 0.004 404 0b825f03c36c2225ab082c2a0bab15f4
When hitting through the proxy most of these requests return 302s rather than 404s. Can the ingress not access these URLs in namespace kube-system?
Am I missing something obvious here - maybe hitting the wrong Kibana URL? I've done a lot of googling and can't find anything similar.
To launch Kibana on Kubernetes, we'll create a Service called kibana , and a Deployment consisting of one Pod replica. You can scale the number of replicas depending on your production needs, and optionally specify a LoadBalancer type for the Service to load balance requests across the Deployment pods.
The Kong Ingress Controller for Kubernetes is an ingress controller driving Kong Gateway. Kusk Gateway is an OpenAPI-driven ingress controller based on Envoy. The NGINX Ingress Controller for Kubernetes works with the NGINX webserver (as a proxy).
An Ingress controller is responsible for fulfilling the Ingress, usually with a load balancer, though it may also configure your edge router or additional frontends to help handle the traffic.
Turns out the problem was with the kibana configuration.
In the kibana deployment yaml there is an environment variable called SERVER_BASEPATH
which is set to point at the kibana service proxy. This was causing the URL to be rewritten each time I tried to access the endpoint externally.
If you comment out this variable and it's value and redeploy kibana then it should work by just hitting the ingress address.
e.g. http://kibana.dev.example1.com/
turns out following environment variables needs to be set in order to expose kibana through ingress:
add following in env in kibana deployment.yaml:
- name: ELASTICSEARCH_HOSTS
value: "http://10.20.30.40:9200"
- name: SERVER_BASEPATH
value: "/kibana"
- name: SERVER_REWRITEBASEPATH
value: "true"
- name: SERVER_PUBLICBASEURL
value: "https://my.domain.com/kibana"
then use following to expose it over ingress:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
namespace: elastic
name: gateway-ingress
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: my.domain.com
http:
paths:
- path: /kibana
backend:
serviceName: kibana
servicePort: 5601
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With