How to properly log the "Path" in K8S ingress-nginx metrics

Question

I'm using ingress-nginx as an Ingress controller for one of my services running over K8S (I'm using the nginx-0.20.0 release image with no specific metrics configurations in the K8S configmap the ingress-controller is using).

The nginx-ingress-controller pods are successfully scraped into my Prometheus server but all ingress metrics (e.g. nginx_ingress_controller_request_duration_seconds_bucket) show up with path="/" regardless of the real path of the handled request.

Worth noting that when I look at the ingress logs - the path is logged correctly.

How can I get the real path noted in the exported metrics?

Thanks!

Answer 1

The Path attribute in the NGINX metrics collected by prometheus derives from the Ingress definition yaml.

For example, if your ingress is:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: <some-k8s-ingress-name>
  namespace: <some-k8s-namespace-name>
spec:
  rules:
  - host: <hostname>
    http:
      paths:
      - backend:
          serviceName: <some-k8s-service-name>
          servicePort: <some-port>
        path: /

Then although NGINX will match any URL to your service, it'll all be logged under the path "/" (as seen here).

If you want metrics for a specific URL, you'll need to explicitly specify it like this (notice the ordering of rules):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 annotations:
   kubernetes.io/ingress.class: nginx
 name: <some-k8s-ingress-name>
 namespace: <some-k8s-namespace-name>
spec:
 rules:
 - host: <hostname>
   http:
     paths:
     - backend:
         serviceName: <some-k8s-service-name>
         servicePort: <some-port>
       path: /more/specific/path
     - backend:
         serviceName: <some-k8s-service-name>
         servicePort: <some-port>
       path: /