Ingress nginx cache

Question

I'm trying to figure out how to use nginx proxy cache with some specific rules. For exemple, when i'm hosting Ghost or Wordpress, I don't want to cache admin section. Using server snippet, I've tried a lot of different combinaison but still have issues with cache in admin section.

nginx.ingress.kubernetes.io/proxy-buffering: "on"
nginx.ingress.kubernetes.io/server-snippet: |-
  proxy_ignore_headers X-Accel-Expires Expires Cache-Control;
  proxy_ignore_headers Set-Cookie;
  proxy_cache app_cache;
  proxy_cache_lock on;
  proxy_cache_valid any 30m;
  add_header X-Cache-Status $upstream_cache_status;

I want to use nginx code snippet for (ghost|sinout) paths to bypass cache when in admin area, but i'm loosing the proxy_pass context resulting to a 502 bad gateway.

Here is the current ingress config caching every pages, admin path too:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-buffering: "on"
    nginx.ingress.kubernetes.io/server-snippet: |-
      proxy_cache my_blog_cache;
      proxy_cache_lock on;
      proxy_cache_valid any 30m;
      add_header X-Cache-Status $upstream_cache_status;
      proxy_ignore_headers X-Accel-Expires Expires Cache-Control;
  name: my-blog
  namespace: web
spec:
  rules:
  - host: blog.example.com
    http:
      paths:
      - backend:
          serviceName: ingress-541322b8660dbd2ceb1e8ff1813f0dd5
          servicePort: 2368
        path: /
  tls:
  - hosts:
    - blog.example.com
    secretName: my-blog-cert
status:
  loadBalancer:
    ingress:
    - ip: 1.2.3.4

Here is the nginx config i'm trying to get but not compatible with ingress annotations:

   location / {
        proxy_cache my_blog_cache;
        proxy_cache_valid 200 30m;
        proxy_cache_valid 404 1m;
        proxy_pass http://ghost_upstream;
        proxy_ignore_headers X-Accel-Expires Expires Cache-Control;
        proxy_ignore_headers Set-Cookie;
        proxy_hide_header Set-Cookie;
        proxy_hide_header X-powered-by;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        expires 10m;
    }
    location /content/images {
        alias /path/to/ghost/content/images;
        access_log off;
        expires max;
    }
    location /assets {
        alias /path/to/ghost/content/themes/uno-master/assets;
        access_log off;
        expires max;
    }
    location /public {
        alias /path/to/ghost/core/built/public;
        access_log off;
        expires max;
    }
    location /ghost/scripts {
        alias /path/to/ghost/core/built/scripts;
        access_log off;
        expires max;
    }
    location ~ ^/(?:ghost|signout) { 
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $http_host;
        proxy_pass http://ghost_upstream;
        add_header Cache-Control "no-cache, private, no-store, must-revalidate, max-stale=0, post-check=0, pre-check=0";
    }

Thanks for your help !

Answer 1

I had exact same desire: cache Ghost responses with respect to their Cache-Control headers directly on Nginx Ingress in the Kubernetes cluster.

After some hours spent on it here is my solution:

First step

First of all, you need to define proxy_cache_path on Nginx Ingress ConfigMap level (the docs are really unclear on how to apply it tbh).

In my case, I manage Nginx Ingress installation via Helm, so I've added it to Helm values chart:

# Default values https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml
controller:
  config:
    http-snippet: "proxy_cache_path /tmp/nginx_my_cache levels=1:2 keys_zone=mycache:2m use_temp_path=off max_size=2g inactive=48h;"

Then apply this change:

helm upgrade -f my-nginx-ingress-values.yaml ingress-nginx ingress-nginx/ingress-nginx --recreate-pods

Second step

Now that we have proxy_cache_path set, we need to configure Ingress for particular host with annotations:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myingress
  namespace: mynamespace
  annotations:
    kubernetes.io/ingress.class: "nginx"
    # Buffering must be enabled for Nginx disk cache to work.
    nginx.ingress.kubernetes.io/proxy-buffering: "on"
    # See https://www.nginx.com/blog/nginx-caching-guide/
    # Cache Key Zone is configured in Helm config.
    nginx.ingress.kubernetes.io/server-snippet: |
      proxy_cache mycache;
      proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
      proxy_cache_background_update on;
      proxy_cache_revalidate on;
      proxy_cache_lock on;
      add_header X-Cache-Status $upstream_cache_status;

Note:

I spent most time figuring out why I was still getting MISSes. Turned out it's due to nginx.ingress.kubernetes.io/proxy-buffering default in Ingress — off — this DISABLES Nginx caching, thus you have to set it to on which is what we do.

Apply the change to Ingress.

Debugging resulting Nginx config

You can and should I think verify resulting nginx.conf used for Ingress that is generated as result of applying ConfigMap and Ingress-level annotations.

To do so, you can copy nginx.conf from Ingress Controller pod to your local machine and verify its content (or exec into pod and see it there):

# Make sure to use correct namespace where Ingress Controller is deployed
# and correct Ingress Controller Pod name
kubectl cp -n default ingress-nginx-controller-xxxx:/etc/nginx/nginx.conf ~/Desktop/nginx.conf

It should contain all the changes we've made!

Debugging actual response caching

Now that we have everything configured — it's time to verify actual caching. Note that we have added X-Cache-Status header which will indicate if it's a HIT or MISS.

I personally like httpie for HTTP requests from Terminal, you can use curl or browser:

First request is going to be MISS:

http https://example.com/myimage.jpg
HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: public, max-age=31536000
Connection: keep-alive
Content-Length: 53588
Content-Type: image/jpeg
Date: Wed, 20 Oct 2021 10:39:06 GMT
ETag: W/"d154-17c3aa43389"
Last-Modified: Fri, 01 Oct 2021 06:56:52 GMT
Strict-Transport-Security: max-age=15724800; includeSubDomains
X-Cache-Status: HIT
X-Powered-By: Express
X-Request-ID: 0c73f97cb51d3071f14968720a26a99a

+-----------------------------------------+
| NOTE: binary data not shown in terminal |
+-----------------------------------------+

Second request to same URL is now a HIT and doesn't hit the actual Ghost installation, success!

http https://example.com/myimage.jpg
HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: public, max-age=31536000
Connection: keep-alive
Content-Length: 53588
Content-Type: image/jpeg
Date: Wed, 20 Oct 2021 10:39:43 GMT
ETag: W/"d154-17c3aa43389"
Last-Modified: Fri, 01 Oct 2021 06:56:52 GMT
Strict-Transport-Security: max-age=15724800; includeSubDomains
X-Cache-Status: HIT
X-Powered-By: Express
X-Request-ID: 0c73f97cb51d3071f14968720a26a99a

+-----------------------------------------+
| NOTE: binary data not shown in terminal |
+-----------------------------------------+

It's also useful to verify logs on Ghost to double-check that cache HIT requests actually served directly from Nginx and never hit Ghost.

---