Connect kibana to elasticsearch in kubernetes cluster

Question

I have a running elasticsearch cluster and I am trying to connect kibana to this cluster (same node). Currently the page hangs when I try to open the service in my browser using :. . In my kibana pod logs, the last few log messages in the pod are:

{"type":"log","@timestamp":"2017-10-13T17:23:46Z","tags":["listening","info"],"pid":1,"message":"Server running at http://0.0.0.0:5601"}
{"type":"log","@timestamp":"2017-10-13T17:23:46Z","tags":["status","ui settings","error"],"pid":1,"state":"red","message":"Status changed from uninitialized to red - Elasticsearch plugin is red","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-10-13T17:23:49Z","tags":["status","plugin:[email protected]","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - Request Timeout after 3000ms","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}

My kibana.yml file that is mounted into the kibana pod has the following config:

server.name: kibana-logging
server.host: 0.0.0.0 
elasticsearch.url: http://elasticsearch:9300
xpack.security.enabled: false
xpack.monitoring.ui.container.elasticsearch.enabled: true 

and my elasticsearch.yml file has the following config settings (I have 3 es pods)

cluster.name: elasticsearch-logs
node.name: ${HOSTNAME}
network.host: 0.0.0.0

bootstrap.memory_lock: false
xpack.security.enabled: false
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts: ["172.17.0.3:9300", "172.17.0.4:9300", "172.17.0.4:9300"]

I feel like the issue is currently with the network.host field but I'm not sure. What fields am I missing/do I need to modify in order to connect to a kibana pod to elasticsearch if they are in the same cluster/node? Thanks!

ES Service:

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  labels:
    component: elasticsearch
    role: master
spec:
  type: NodePort
  selector:
    component: elasticsearch
    role: master
  ports:
  - name: http
    port: 9200
    targetPort: 9200
    nodePort: 30303
    protocol: TCP

Kibana Svc

apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: default
  labels:
    component: kibana
spec:
  type: NodePort
  selector:
    component: kibana
  ports:
  - port: 80
    targetPort: 5601
    protocol: TCP

EDIT: After changing port to 9200 in kibana.yml here is what i see in the logs at the end when I try and access kibana:

{"type":"log","@timestamp":"2017-10-13T21:36:30Z","tags":["listening","info"],"pid":1,"message":"Server running at http://0.0.0.0:5601"}
{"type":"log","@timestamp":"2017-10-13T21:36:30Z","tags":["status","ui settings","error"],"pid":1,"state":"red","message":"Status changed from uninitialized to red - Elasticsearch plugin is red","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-10-13T21:36:33Z","tags":["status","plugin:[email protected]","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - Request Timeout after 3000ms","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2017-10-13T21:37:02Z","tags":["error","elasticsearch","admin"],"pid":1,"message":"Request error, retrying\nPOST http://elasticsearch:9200/.reporting-*/esqueue/_search?version=true => getaddrinfo ENOTFOUND elasticsearch elasticsearch:9200"}
{"type":"log","@timestamp":"2017-10-13T21:37:32Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2017-10-13T21:37:33Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2017-10-13T21:37:37Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2017-10-13T21:37:38Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2017-10-13T21:37:42Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}

Answer 1

The issue here is that you exposed Elasticsearch on port 9200 but are trying to connect to port 9300 in your kibana.yml file.

You either need to edit your kibana.yml file to use:

elasticsearch.url: http://elasticsearch:9200

Or change the port in the elasticsearch service to 9300.