ServiceMonitor Prometheus not working in another namespace

Question

I just installed prometheus operator as indicated here: https://github.com/prometheus-operator/kube-prometheus:

kubectl apply --server-side -f manifests/setup
kubectl wait \
    --for condition=Established \
    --all CustomResourceDefinition \
    --namespace=monitoring
kubectl apply -f manifests/

After that I just tried to setup my own service monitor for grafana as follows:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: in1-grafana-service-monitor
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: grafana
  endpoints:
  - port: http
    interval: 10s

This monitor works just fine and I can see it in the Prometheus /targets and /service-discovery.

The fact is that when I want to create this same service monitor but outside the "monitoring" namespace it just not appears neither in /targets or in /service-discovery. My setup for this service monitor is as follows:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: out1-grafana-service-monitor
  namespace: other-namespace
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: grafana
  namespaceSelector:
    any: true
  endpoints:
  - port: http
    interval: 10s

How can I make Prometheus operator to scrape service monitors (and services) outside the monitoring namespace?

I checked the output of kubectl get prom -Ao yaml and it just displays an empty list:

[...]
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector: {}
[...]

Any help will be appreciated.

Thank you.

I expect that the service monitor outside the monitoring namespace works as I need it for other service (Not for Grafana).

Answer 1

After looking at the yaml files I realized that Prometheus doesn't have the permissions to read all namespaces. And after looking at the repository customization examples I found the solution: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizations/monitoring-additional-namespaces.md

Hope this helps someone else in the future.

Answer 2

Based on this Prometheus code, you can do something like:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus-k8s
  namespace: monitoring
spec:
  serviceMonitorNamespaceSelector:
    matchNames:
      - monitoring
      - your-namespace # Add your additional namespace here
  serviceMonitorSelector:
    matchLabels:
      team: frontend # Define match labels to select appropriate ServiceMonitors
  podMonitorNamespaceSelector:
    matchNames:
      - monitoring
      - your-namespace # Add your additional namespace here
  podMonitorSelector:
    matchLabels:
      team: frontend # Define match labels to select appropriate PodMonitors
  # Other spec fields...