nginx proxyPass to prometheus using variable

Question

I have the following nginx.conf

location /monitoring/prometheus/ {
  resolver 172.20.0.10 valid=5s;
  set $prometheusUrl http://prometheus.monitoring.svc.cluster.local:9090/;

  proxy_set_header Accept-Encoding "";
  proxy_pass $prometheusUrl;
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;

  sub_filter_types text/html;
  sub_filter_once off;
  sub_filter '="/' '="/monitoring/prometheus/';
  sub_filter 'var PATH_PREFIX = "";' 'var PATH_PREFIX = "/monitoring/prometheus";';

  rewrite ^/monitoring/prometheus/?$ /monitoring/prometheus/graph redirect;
  rewrite ^/monitoring/prometheus/(.*)$ /$1 break;
}

When I naviagte to https://myHost/monitoring/prometheus/graph I get redirected to /graph (https://myHost/graph)

When I don't use the variable and place the url directly to proxy_pass everything works as expected. I can navigate to https://myHost/monitoring/prometheus/graph and see prometheus.

location /monitoring/prometheus/ {
  resolver 172.20.0.10 valid=5s;

  proxy_set_header Accept-Encoding "";
  proxy_pass http://prometheus.monitoring.svc.cluster.local:9090/;
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;

  sub_filter_types text/html;
  sub_filter_once off;
  sub_filter '="/' '="/monitoring/prometheus/';
  sub_filter 'var PATH_PREFIX = "";' 'var PATH_PREFIX = "/monitoring/prometheus";';

  rewrite ^/monitoring/prometheus/?$ /monitoring/prometheus/graph redirect;
  rewrite ^/monitoring/prometheus/(.*)$ /$1 break;
}

Can anyone explain to me why using the variable leads to a different behaviour in terms of routing? I need to use variables to force nginx to resolve the dns name on each request.

Answer 1

I just figured it out. As stated in the docs

When variables are used in proxy_pass:

location /name/ {
    proxy_pass http://127.0.0.1$request_uri;
}

In this case, if URI is specified in the directive, it is passed to the server as is, replacing the original request URI.

So the problem was that I specified a request uri in the variable (the trailing /). After removing this / everything worked fine.

Here the working config:

location /monitoring/prometheus/ {
  set $prometheusUrl http://prometheus.monitoring.svc.cluster.local:9090;

  proxy_set_header Accept-Encoding "";
  proxy_pass $prometheusUrl;
  proxy_redirect off;
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;

  sub_filter_types text/html;
  sub_filter_once off;
  sub_filter '="/' '="/monitoring/prometheus/';
  sub_filter 'var PATH_PREFIX = "";' 'var PATH_PREFIX = "/monitoring/prometheus";';

  rewrite ^/monitoring/prometheus/?$ /monitoring/prometheus/graph redirect;
  rewrite ^/monitoring/prometheus/(.*)$ /$1 break;
}

Answer 2

I believe it's best to use the prometheus --web.external-url command line option.

This sandbox example shows a full working stack, complete with nginx config. https://github.com/prometheus-community/prometheus-playground/tree/master/nginx

Of note is the prometheus command section of the docker-compose.yml file.

command:
- --config.file=/etc/prometheus/prometheus.yml
- --web.route-prefix=/
- --web.external-url=http://example.com/prometheus