How do I access this Kubernetes service via kubectl proxy?

Question

I want to access my Grafana Kubernetes service via the kubectl proxy server, but for some reason it won't work even though I can make it work for other services. Given the below service definition, why is it not available on http://localhost:8001/api/v1/proxy/namespaces/monitoring/services/grafana?

grafana-service.yaml

apiVersion: v1
kind: Service
metadata:
  namespace: monitoring
  name: grafana
  labels:
    app: grafana
spec:
  type: NodePort
  ports:
  - name: web
    port: 3000
    protocol: TCP
    nodePort: 30902
  selector:
    app: grafana

grafana-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  namespace: monitoring
  name: grafana
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:4.1.1
        env:
        - name: GF_AUTH_BASIC_ENABLED
          value: "true"
        - name: GF_AUTH_ANONYMOUS_ENABLED
          value: "true"
        - name: GF_SECURITY_ADMIN_USER
          valueFrom:
            secretKeyRef:
              name: grafana-credentials
              key: user
        - name: GF_SECURITY_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              name: grafana-credentials
              key: password
        volumeMounts:
        - name: grafana-storage
          mountPath: /var/grafana-storage
        ports:
        - name: web
          containerPort: 3000
        resources:
          requests:
            memory: 100Mi
            cpu: 100m
          limits:
            memory: 200Mi
            cpu: 200m
      - name: grafana-watcher
        image: quay.io/coreos/grafana-watcher:v0.0.5
        args:
          - '--watch-dir=/var/grafana-dashboards'
          - '--grafana-url=http://localhost:3000'
        env:
        - name: GRAFANA_USER
          valueFrom:
            secretKeyRef:
              name: grafana-credentials
              key: user
        - name: GRAFANA_PASSWORD
          valueFrom:
            secretKeyRef:
              name: grafana-credentials
              key: password
        resources:
          requests:
            memory: "16Mi"
            cpu: "50m"
          limits:
            memory: "32Mi"
            cpu: "100m"
        volumeMounts:
        - name: grafana-dashboards
          mountPath: /var/grafana-dashboards
      volumes:
      - name: grafana-storage
        emptyDir: {}
      - name: grafana-dashboards
        configMap:
          name: grafana-dashboards

The error I'm seeing when accessing the above URL is "no endpoints available for service "grafana"", error code 503.

Answer 1

As Michael says, quite possibly your labels or namespaces are mismatching. However in addition to that, keep in mind that even when you fix the endpoint, the url you're after (http://localhost:8001/api/v1/proxy/namespaces/monitoring/services/grafana) might not work correctly.

Depending on your root_url and/or static_root_path grafana configuration settings, when trying to login you might get grafana trying to POST to http://localhost:8001/login and get a 404.

Try using kubectl port-forward instead:

kubectl -n monitoring port-forward [grafana-pod-name] 3000

then access grafana via http://localhost:3000/

https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/