Using Traefik to reverse proxy Grafana at a suburl; 404 responses

Question

I have tried my best to follow the documentation for both Grafana and Traefik to reverse proxy incoming requests to https://metrics.mydomain/grafana through Traefik to a Grafana instance running as a container, but I get 404 responses from Grafana and I can't figure out why.

Here's what I've done after reading docs and trying various things in my docker-compose.yml file:

version: '3'
services:
  traefik:
    image: traefik
    restart: always
    ports:
     - 80:80
     - 443:443
     - 8080:8080
    volumes:
     - /var/run/docker.sock:/var/run/docker.sock
     - /home/ben/traefik.toml:/etc/traefik/traefik.toml
     - /etc/letsencrypt/live/mydomain/fullchain.pem:/mydomain_fullchain.pem
     - /etc/letsencrypt/live/mydomain/privkey.pem:/mydomain_privkey.pem
  grafana:
    image: grafana/grafana
    restart: always
    ports:
     - 3000:3000
    volumes:
     - /srv/grafana:/var/lib/grafana
    environment:
     - GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s:%(http_port)s/grafana
    labels:
     - traefik.enable=true
     - 'traefik.frontend.rule=Host:metrics.mydomain;PathPrefix:/grafana;'
     - 'traefik.frontend.redirect.entryPoint=https'

Traefik does pick up on this. Here's what it shows on the dashboard:

However, when I actually request https://metrics.mydomain/grafana, what I get back is 404 and an error document, and then subsequent 404s for the resources requested by the custom error document:

Finally, and most confusingly, it appears that the requests as received by Grafana itself are correct, but still result in 404s:

method=GET path=/grafana status=404 remote_addr=192.168.0.1
method=GET path=/grafana/public/build/grafana.dark.css status=404 remote_addr=192.168.0.1
...etc

I don't understand why this is so fussy, and there doesn't appear (to me) to be any evidence that would suggest what the actual problem is.

p.s. - I did leave a comment on a Grafana issue that looked relevant. https://github.com/grafana/grafana/issues/2066#issuecomment-380647174

Answer 1

I was able to fix this by using PathPrefixStrip instead of PathPrefix