Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to update Prometheus config in k8s cluster

I have running Prometheus in k8s. Could you please advice how can I change running config prometheus.yaml in cluster? I just want simply to change:

scrape_configs:
- job_name: my-exporter
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http

How can I do this?

Thanks.

like image 461
JBB Avatar asked Nov 09 '18 14:11

JBB


People also ask

How do I update Prometheus configuration in Kubernetes?

You can either do that manually by sending a POST request as described in the link above, or automate this process by having a sidecar container inside the same prometheus pod that watch for updates to the config file and does the reload POST request.

How do I update Prometheus config?

Prometheus can reload its configuration at runtime. If the new configuration is not well-formed, the changes will not be applied. A configuration reload is triggered by sending a SIGHUP to the Prometheus process or sending a HTTP POST request to the /-/reload endpoint (when the --web. enable-lifecycle flag is enabled).


1 Answers

The recommended way is to provide the prometheus.yml via a ConfigMap. That way changes in the ConfigMap will be propagated into the pod that consumes the configMap. However, that is not enough for prometheus to pick up the new config.

Prometheus supports runtime reload of the config, so that you don't need to stop prometheus in order to pickup the new config. You can either do that manually by sending a POST request as described in the link above, or automate this process by having a sidecar container inside the same prometheus pod that watch for updates to the config file and does the reload POST request.

The following is an example on the second approach: prometheus-configmaps-continuous-deployment

like image 167
yamenk Avatar answered Sep 21 '22 03:09

yamenk