Forbidden resource in API group at the cluster scope

Question

I am unable to identify what the exact issue with the permissions with my setup as shown below. I've looked into all the similar QAs but still unable to solve the issue. The aim is to deploy Prometheus and let it scrape /metrics endpoints that my other applications in the cluster expose fine.

Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"endpoints\" in API group \"\" at the cluster scope"
Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"services\" in API group \"\" at the cluster scope"
...
...

The command below returns no to all services, nodes, pods etc.

kubectl auth can-i get services --as=system:serviceaccount:default:default -n default

Minikube

$ minikube start --vm-driver=virtualbox --extra-config=apiserver.Authorization.Mode=RBAC

😄  minikube v1.14.2 on Darwin 11.2
✨  Using the virtualbox driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🔄  Restarting existing virtualbox VM for "minikube" ...
🐳  Preparing Kubernetes v1.19.2 on Docker 19.03.12 ...
    ▪ apiserver.Authorization.Mode=RBAC
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass, dashboard
🏄  Done! kubectl is now configured to use "minikube" by default

Roles

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole

metadata:
  name: monitoring-cluster-role

rules:
  - apiGroups: [""]
    resources: ["nodes", "services", "pods", "endpoints"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get"]
  - apiGroups: ["extensions"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch"]

apiVersion: v1
kind: ServiceAccount

metadata:
  name: monitoring-service-account
  namespace: default

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding

metadata:
  name: monitoring-cluster-role-binding

roleRef:
  kind: ClusterRole
  name: monitoring-cluster-role
  apiGroup: rbac.authorization.k8s.io

subjects:
  - kind: ServiceAccount
    name: monitoring-service-account
    namespace: default

Prometheus

apiVersion: v1
kind: ConfigMap
 
metadata:
  name: prometheus-config-map
  namespace: default
 
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
    scrape_configs:
      - job_name: 'kubernetes-service-endpoints'
        kubernetes_sd_configs:
        - role: endpoints
        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_service_name]
          action: replace
          target_label: kubernetes_name 

apiVersion: apps/v1
kind: Deployment
 
metadata:
  name: prometheus-deployment
  namespace: default
  labels:
    app: prometheus
 
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus:latest
          ports:
            - name: http
              protocol: TCP
              containerPort: 9090
          volumeMounts:
            - name: config
              mountPath: /etc/prometheus/
            - name: storage
              mountPath: /prometheus/
      volumes:
        - name: config
          configMap:
            name: prometheus-config-map
        - name: storage
          emptyDir: {}

apiVersion: v1
kind: Service
 
metadata:
  name: prometheus-service
  namespace: default
 
spec:
  type: NodePort
  selector:
    app: prometheus
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 9090

Answer 1

User "system:serviceaccount:default:default" cannot list resource "endpoints" in API group "" at the cluster scope"

User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" at the cluster scope"

User "system:serviceaccount:default:default" cannot list resource "services" in API group "" at the cluster scope"

Something running with ServiceAccount default in namespace default is doing things it does not have permissions for.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: monitoring-service-account

Here you create a specific ServiceAccount. You also give it some Cluster-wide permissions.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-deployment
  namespace: default

You run Prometheus in namespace default but do not specify a specific ServiceAccount, so it will run with ServiceAccount default.

I think your problem is that you are supposed to set the ServiceAccount that you create in the Deployment-manifest for Prometheus.