nginx proxy_pass leads to 404 Not Found page

Question

I have a angular application running in a docker ubuntu image that has nginx installed. I want to deploy this image to Kubernetes and use a nginx proxy to redirect all calls to /api to my backend service in Kubernetes.

My static web resources lie in /var/www/html and I add the following config to /etc/nginx/conf.d:

upstream backend-service {
  server backend-service:8080;
}

server {
  listen 80;

  location / {
    try_files $uri $uri/ /index.html;
  }

  location ^~ /api {
    proxy_pass http://backend-service;
  }
}

Accessing the frontend service on / or /#/dashboard returns the expected component of my Angular page, but a call to /api/v1/data only shows the default nginx 404 Not Found page.

What do I need to modify to have my backend calls redirected to my backend?

I use nginx 1.10.3 on ubuntu 16.04 and my frontend Dockerfile looks like this:

FROM ubuntu:16.04

# Install curl, nodejs and nginx
RUN apt-get update && \
  apt-get install -y curl && \
  curl -sL https://deb.nodesource.com/setup_8.x | bash - && \
  apt-get install -y nodejs nginx && \
  rm -rf /var/lib/apt/lists/*

# Create directory
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

# Copy and build rest of the app
COPY . /usr/src/app
RUN npm install
RUN node_modules/@angular/cli/bin/ng build --prod
RUN cp -a dist/. /var/www/html

# Configure and start nginx
COPY frontend.conf /etc/nginx/conf.d

EXPOSE 80

CMD ["nginx", "-g", "daemon off;"]

Edit: Information about backend-service

The backend service listens to get and post requests on /api/v1/data and is reachable in Kubernetes via a Service named backend-service.

Edit2: Nginx access.log

https://gist.github.com/Steffen911/a56e3175bf12e511048d01359a475724

172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET / HTTP/1.1" 200 380 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /styles.d41d8cd98f00b204e980.bundle.css HTTP/1.1" 200 0 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /inline.c9a1a6b995c65c13f605.bundle.js HTTP/1.1" 200 1447 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /polyfills.117078cae3e3d00fc376.bundle.js HTTP/1.1" 200 97253 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /main.3e9a37b4dd0f3bf2465f.bundle.js HTTP/1.1" 200 64481 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /vendor.146173c1a99cc2172a5f.bundle.js HTTP/1.1" 200 661261 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /api/v1/data/ HTTP/1.1" 404 209 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /assets/home.jpg HTTP/1.1" 200 2608 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /assets/busy.gif HTTP/1.1" 200 48552 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /assets/background_light.png HTTP/1.1" 200 170599 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /assets/google.svg HTTP/1.1" 200 2232 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /assets/email.svg HTTP/1.1" 200 1596 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /favicon.ico HTTP/1.1" 200 198 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:44 +0000] "GET /api/v1/data/ HTTP/1.1" 404 209 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"

The error.log file is empty.

Edit3: New nginx version and other SO thread

I also tried nginx 1.12.1 and it shows the same behaviour. The answers to this question also haven't helped: nginx proxy_pass 404 error, don't understand why

Edit4: I uploaded a minimal example that reproduces my problem on GitHub

https://github.com/Steffen911/nginx-sample

Answer 1

From your troubleshooting of nginx, it appears that the nginx configuration file you have has effectively no effect — you report getting 404 Not Found errors for everything other than the index page, with all the directives, from try_files in location /, to proxy_pass and return 200 test in a more specific location ^~ /api, having no effect.

As such, the problem appears to be in the Dockerfile — it appears that most other NGINX + Docker tutorials remove default configurations (e.g., with RUN rm /etc/nginx/conf.d/default.conf), whereas your file is missing any such removal.

In fact, Debian/Ubuntu appear to have the non-standard directories of questionable utility called /etc/nginx/sites-available and /etc/nginx/sites-enabled, which, by default, must contain a default file with a presumptuous listen 80 default_server, effectively taking precedence over any other listen of the same port in the absence of a more specific server_name.

---

As such, there are multiple independent solutions:

---

• Do not use fundamentally broken packages like those offered by Debian/Ubuntu. I once spent a good amount of time pulling my hair trying to figure out why my configs don't work, only to notice that even the backup files from emacs like test.conf~ get included in Debian through Debian's default include /etc/nginx/sites-enabled/*;. Sites-Enabled Is Evil.

  Note that NGINX provides official binary packages for most distributions, which wouldn't have had this issue, as it doesn't try to define a default_server in its /etc/nginx/conf.d/default.conf, instead doing a listen 80; with server_name localhost;, getting out of your way automatically by itself in most circumstances.

  E.g., replace FROM ubuntu:16.04 with FROM nginx in your Dockerfile to be using NGINX official image.

---

• If still using nginx from Debian/Ubuntu, make sure to RUN rm /etc/nginx/sites-enabled/default in your Dockerfile to remove the default_server listen.

---

• Use the server_name directive to define hostname-based servers, possibly together with the listen directive with the default_server parameter, too.

  Note that a duplicate server_name specification results in a configuration warning (with the [warn] severity), but a duplicate default_server is a configuration error ([emerg] severity), which might help troubleshoot the issue earlier.