Kubernetes Ingress path rewrites

Question

I am running Ingress for some services on Kubernetes, and although services are re-directing to those cluster IPs (somewhat correctly), I am not sure how to correctly resolve paths after just those basic paths, for example, if I have Tomcat in path / and Nginx on path /nginx, any Tomcat path after / does not resolve and neither does any Nginx path resolve for even /. I already have nginx.ingress.kubernetes.io/rewrite-target: / set in my ingress.yaml:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: tomcat-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
  namespace: kube-system
spec:
  rules:
  - http:
      paths:
      - path: /
        backend:
          serviceName: tomcat-deployment-service 
          servicePort: 8080
      - path: /nginx
        backend:
          serviceName: nginx-deployment-service 
          servicePort: 80

If Tomcat requires a /main.css file, for example, the browser would try to fetch /main.css, but this returns the default-backend.

Also, if I try to visit /nginx (attempting to hit /) on the Nginx server, Nginx says /nginx is not found (obviously).

2019/03/02 08:12:04 [error] 8#8: *343 open() "/usr/share/nginx/html/nginx" failed (2: No such file or directory), client: 10.128.0.7, server: localhost, request: "GET /nginx HTTP/1.1", host: "REDACTED_SERVER_IP"
10.128.0.7 - - [02/Mar/2019:08:12:04 +0000] "GET /nginx HTTP/1.1" 404 153 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:65.0) Gecko/20100101 Firefox/65.0" "REDACTED_CLIENT_IP, REDACTED_SERVER_IP"
10.40.1.1 - - [02/Mar/2019:08:12:05 +0000] "GET /nginx HTTP/1.1" 404 153 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:65.0) Gecko/20100101 Firefox/65.0" "REDACTED_CLIENT_IP, REDACTED_SERVER_IP"
2019/03/02 08:12:05 [error] 8#8: *344 open() "/usr/share/nginx/html/nginx" failed (2: No such file or directory), client: 10.40.1.1, server: localhost, request: "GET /nginx HTTP/1.1", host: "REDACTED_SERVER_IP"

How can I make paths after a specified path in ingress.yaml resolve considering the beginning part of the path as viewed from the service?

i.e.

1. If Tomcat was ingress pathed to /tomcat, how could I make /tomcat/main.css appear as /main.css to the Tomcat server (and not resolve to default backed)?
2. If Nginx was ingress pathed to /nginx, how could I make /nginx appear as / to the Nginx server (and not appear as /nginx)?

I thought this was what /rewrite-target was suppose to do.

Would I need to use wildcards or something?

My ingress is the following:

Name:             tomcat-ingress
Namespace:        kube-system
Address:          REDACTED_SERVER_IP
Default backend:  default-http-backend:80 (10.40.0.6:8080)
Rules:
  Host  Path  Backends
  ----  ----  --------
  *         
        /        tomcat-deployment-service:8080 (10.40.2.15:8080)
        /nginx   nginx-dep-ser:80 (10.40.0.26:80,10.40.1.46:80)
Annotations:
  ingress.kubernetes.io/url-map:                     k8s-um-kube-system-tomcat-ingress--b0fc8aa23db1001d
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"nginx.ingress.kubernetes.io/rewrite-target":"/"},"name":"tomcat-ingress","namespace":"kube-system"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"tomcat-deployment-service","servicePort":8080},"path":"/"},{"backend":{"serviceName":"nginx-dep-ser","servicePort":80},"path":"/nginx"}]}}]}}

  nginx.ingress.kubernetes.io/rewrite-target:  /
  ingress.kubernetes.io/backends:              {"k8s-be-30985--b0fc8aa23db1001d":"HEALTHY","k8s-be-31229--b0fc8aa23db1001d":"HEALTHY","k8s-be-32736--b0fc8aa23db1001d":"HEALTHY"}
  ingress.kubernetes.io/forwarding-rule:       k8s-fw-kube-system-tomcat-ingress--b0fc8aa23db1001d
  ingress.kubernetes.io/target-proxy:          k8s-tp-kube-system-tomcat-ingress--b0fc8aa23db1001d
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  CREATE  33m                nginx-ingress-controller  Ingress kube-system/tomcat-ingress
  Normal  ADD     33m                loadbalancer-controller   kube-system/tomcat-ingress
  Normal  UPDATE  3m (x65 over 33m)  nginx-ingress-controller  Ingress kube-system/tomcat-ingress
  Normal  CREATE  3m (x31 over 33m)  loadbalancer-controller   ip: REDACTED_SERVER_IP

Answer 1

I assume that you have quite recent version of ingress controller, and according to the documentation:

Starting in Version 0.22.0, ingress definitions using the annotation nginx.ingress.kubernetes.io/rewrite-target are not backwards compatible with previous versions. In Version 0.22.0 and beyond, any substrings within the request URI that need to be passed to the rewritten path must explicitly be defined in a capture group.

E.g:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
  name: tomcat-ingress
  namespace: default 
spec:
  rules:
  - host: rewrite.bar.com
    http:
      paths:
      - backend:
          serviceName: tomcat-deployment-service
          servicePort: 8080
        path: /tomcat/?(.*)
      - backend:
          serviceName: nginx-deployment-service
          servicePort: 80
        path: /nginx/?(.*)

Your current configuration works well for / and /nginx destinations only with the recent version of ingress controller.

Another important thing, the Ingress object should be created in the same namespace with the service(s) it refers to. Otherwise ingress-nginx usually returns 503 Service Temporarily Unavailable error.

And it's not allowed to refer to a service in the different namespace in the serviceName:

a DNS-1035 label must consist of lower case alphanumeric characters or '-', start with an alphabetic character, and end with an alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for validation is 'a-z?')