All Kubernetes proxy targets down - Prometheus Operator

Question

I have a k8s cluster deployed in openstack. I have deployed Prometheus operator for it to monitor the cluster. But I am getting Kubernetes proxy down alert for all the nodes.

I would like to know basics of how Prometheus operator scrapes Kubernetes proxy? also would like to know what configurations needs to be done to fix it.

I can see that kube proxy is running in all nodes at 10249 port.

Error :

Get http://10.8.10.11:10249/metrics: dial tcp 10.8.10.11:10249: connect: connection refused

HELM values configuration

kubeProxy:
  enabled: true

  ## If your kube proxy is not deployed as a pod, specify IPs it can be found on
  ##
  endpoints: []
  # - 10.141.4.22
  # - 10.141.4.23
  # - 10.141.4.24

  service:
    port: 10249
    targetPort: 10249
    # selector:
    #   k8s-app: kube-proxy

  serviceMonitor:
    ## Scrape interval. If not set, the Prometheus default scrape interval is used.
    ##
    interval: ""

    ## Enable scraping kube-proxy over https.
    ## Requires proper certs (not self-signed) and delegated authentication/authorization checks
    ##
    https: false

Answer 1

Set the kube-proxy argument for metric-bind-address

$ kubectl edit cm/kube-proxy -n kube-system

...
kind: KubeProxyConfiguration
metricsBindAddress: 0.0.0.0:10249
...

$ kubectl delete pod -l k8s-app=kube-proxy -n kube-system