Prometheus error - Error on ingesting samples

Question

I am getting following error in Prometheus on monitoring my Node JS based web application.

level=warn ts=2019-08-02T18:23:48.658364708Z caller=scrape.go:932 component="scrape manager" scrape_pool=batch_web target=https://example.com:443/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=6

Any pointers on why this could be happening and how can I solve this?

Answer 1

• Enable debugging mode in Prometheus by passing --log.level=debug as an argument to prometheus

• Check the log. In my case the metric that was dropped, was returned:

  msg="Out of order sample" series="some-metric{a="b",c="d"}
  

• Check the prometheus configuration and rule files that could return a duplicated some-metric. In my case a duplicated rule was causing the issue as the same metric was scraped from the federated prometheus and calculated again by my prometheus. Removing the duplicated rule solved the issue.