Context Deadline Exceeded - prometheus

Question

I have Prometheus configuration with many jobs where I am scraping metrics over HTTP. But I have one job where I need to scrape the metrics over HTTPS.

When I access:

https://ip-address:port/metrics

I can see the metrics. The job that I have added in the prometheus.yml configuration is:

- job_name: 'test-jvm-metrics'     scheme: https     static_configs:       - targets: ['ip:port'] 

When I restart the Prometheus I can see an error on my target that says:

context deadline exceeded

I have read that maybe the scrape_timeout is the problem, but I have set it to 50 sec and still the same problem.

What can cause this problem and how to fix it? Thank you!

Answer 1

Probably the default scrape_timeout value is too short for you

[ scrape_timeout: <duration> | default = 10s ] 

Set a bigger value for scrape_timeout.

scrape_configs:   - job_name: 'prometheus'      scrape_interval: 5m     scrape_timeout: 1m 

Take a look here https://github.com/prometheus/prometheus/issues/1438

Answer 2

I had a same problem in the past. In my case the problem was with the certificates and I fixed it with adding:

 tls_config:       insecure_skip_verify: true 

You can try it, maybe it will work.