How to enable additional configs for prometheus operator

Question

according to Prometheus-operator documentation we should be able to supply our additional configuration easily via secret file. Does anybody actually succeed this step? I have several questions:

• where these configuration will appear in prometheus pod?
• should this configuration be in a form of prometheus configuration file or just list additional scrape entries
• can we supply additional files (json configs) via file_sd_configs: and if so how to supply those files into prometheus manifest file?

Regardless of those questions I have hard time to add the additional configuration. I basically followed exact steps from documentation, and here is my observations:

1. here is my new configuration

    cat prometheus-additional.yaml
    - job_name: "prometheus-custom"
      static_configs:
      - targets: ["localhost:9090"]

2. add new secret file

    kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml

3. create prometheus.yml file with additional configuration

    apiVersion: monitoring.coreos.com/v1
    kind: Prometheus
    metadata:
      name: prometheus
    spec:
      replicas: 2
      resources:
        requests:
          memory: 400Mi
      additionalScrapeConfigs:
        name: additional-scrape-configs
        key: prometheus-additional.yaml

4. deploy prometheus.yaml

    kubectl apply -f prometheus.yaml

5. check logs and there is no indication of my new configuration

kubectl logs prometheus-prometheus-0 -c prometheus
level=info ts=2019-12-05T18:07:30.217852541Z caller=main.go:302 msg="Starting Prometheus" version=" (version=2.7.1, branch=HEAD, revision=62e591f928ddf6b3468308b7ac1de1c63aa7fcf3)"
level=info ts=2019-12-05T18:07:30.217916972Z caller=main.go:303 build_context="(go=go1.11.5, user=root@f9f82868fc43, date=20190131-11:16:59)"
level=info ts=2019-12-05T18:07:30.217971648Z caller=main.go:304 host_details="(Linux 4.19.3-300.fc29.x86_64 #1 SMP Wed Nov 21 15:27:25 UTC 2018 x86_64 prometheus-prometheus-0 (none))"
level=info ts=2019-12-05T18:07:30.217994128Z caller=main.go:305 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2019-12-05T18:07:30.218236509Z caller=main.go:306 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2019-12-05T18:07:30.219359123Z caller=main.go:620 msg="Starting TSDB ..."
level=info ts=2019-12-05T18:07:30.219487263Z caller=web.go:416 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2019-12-05T18:07:30.230944675Z caller=main.go:635 msg="TSDB started"
level=info ts=2019-12-05T18:07:30.231037536Z caller=main.go:695 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
level=info ts=2019-12-05T18:07:30.23125837Z caller=main.go:722 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
level=info ts=2019-12-05T18:07:30.231294106Z caller=main.go:589 msg="Server is ready to receive web requests."
level=info ts=2019-12-05T18:07:33.568068248Z caller=main.go:695 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
level=info ts=2019-12-05T18:07:33.568305994Z caller=main.go:722 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml

And, when I logged into prometheus pod I don't see any additional configuration either, and when I check my prometheus web console I don't see any of my configurations.

Answer 1

Turns out, the prometheus-operator still relies on serviceMonitorSelector: {} part of the manifest file according to this ticket. Therefore in order to add additional configuration we need to have the following manifest:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
spec:
  replicas: 2
  resources:
    requests:
      memory: 400Mi
  additionalScrapeConfigs:
    name: prometheus-config
    key: prometheus-config.yaml
  serviceMonitorSelector: {}

where prometheus-config.yaml will contain prometheus scrape rules and deployed via secrets to prometheus cluster. I also found empirically that current prometheus-operator does not support file_sd_configs in prometheus configuration (sad) and someone need to write full rules in prometheus-config.yaml file.