prometheus operator - enable monitoring for everything in all namespaces

Question

I want to monitor a couple applications running on a Kubernetes cluster in namespaces named development and production through prometheus-operator.

Installation command used (as per Github) is:

helm install prometheus-operator stable/prometheus-operator -n production --set prometheusOperator.enabled=true,prometheus.service.type=NodePort,prometheusOperator.service.type=NodePort,alertmanager.service.type=NodePort,grafana.service.type=NodePort,grafana.service.nodePort=30906

What parameters do I need to add to above command to have prometheus-operator discover and monitor all apps/services/pods running in all namespaces?

With this, Service Discovery only shows some prometheus-operator related services, but not the app that I am running within 'production' namespace even though prometheus-operator is installed in the same namespace.

Anything I am missing?

Note - Am running performing all actions using the same user (which uses the $HOME/.kube/config file), so I assume permissions are not an issue.

kubectl version - v1.17.3 helm version - 3.1.2

P.S. There are numerous articles on this on different forums, but am still not finding simple and direct answers for this.

Answer 1

I had the same problem. After some investigation answering with more details.

I've installed Prometheus stack via Helm charts which include Prometheus operator chart directly as a sub-project. Prometheus operator monitors namespaces specified by the following helm values:

prometheusOperator:
  namespaces: ''
  denyNamespaces: ''
  prometheusInstanceNamespaces: ''
  alertmanagerInstanceNamespaces: ''
  thanosRulerInstanceNamespaces: ''

The namespaces value specifies monitored namespaces for ServiceMonitor and PodMonitor CRDs. Other CRDs have their own settings, which if not set, default to namespaces. Helm values are passed as command-line arguments to the operator. See here and here.

Prometheus CRDs are picked up by the operator from the mentioned namespaces, by default - everywhere. However, as the operator is designed with multiple simultaneous Prometheus releases in mind, what to pick up by a particular Prometheus app instance is controlled by the corresponding Prometheus CRD. CRDs selectors and corresponding namespaces selectors are controlled via the following Helm values:

prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: true
    serviceMonitorSelector: {}
    serviceMonitorNamespaceSelector: {}

Similar values are present for other CRDs: alertmanagerConfigXXX, ruleNamespaceXXX, podMonitorXXX, probeXXX. XXXSelectorNilUsesHelmValues set to true, means to look for CRD with particular release label, e.g. release=myrelease. See here.

Empty selector (for a namespace, CRD, or any other object) means no filtering. So for Prometheus object to pick up a ServiceMonitor from the other namespaces there are few options:

• Set serviceMonitorSelectorNilUsesHelmValues: false. This leaves serviceMonitorSelector empty.
• Apply the release label, e.g. release=myrelease, to your ServiceMonitor CRD.
• Set a non-empty serviceMonitorSelector that matches your ServiceMonitor.

For the curious ones here are links to the operator sources:

• Enqueue of Prometheus CRD processing
• Processing of Prometheus CRD