Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

nginx-ingress config map snippets being ignored by the nginx.conf

I have a kubernetes cluster, where I have deployed an nginx ingress controller using the helm nginx-ingress chart.

I need to add some custom config to the nginx.conf file that is generated in the nginx-controller-pod, and I am seeing an issue where if I add a one line option such as proxy-buffer-size: "512k" I can see this reflected in the nginx.conf file and everything works as expected.

However, if I attempt to add a snippet to accomplish the same thing:

location-snippet: |
  proxy_buffer_size "512k";

It is as though this is ignored by the nginx.conf file and the proxy_buffer_size setting remains at it's default value.

I need to be able to add http-snippet, server-snippet and location-snippet overrides but whether I try to add them to the ConfigMap or as an annotation in the Ingress.yaml file they are always ignored.

My Ingress yaml file:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    ingress.kubernetes.io/ssl-redirect: "true" 
    ingress.kubernetes.io/secure-backends: "true"    
    ingress.kubernetes.io/force-ssl-redirect: "true"

    ingress.kubernetes.io/location-snippet: |
       proxy_buffer_size 512k;     --This does not update the nginx.conf
spec:
  tls:
  - hosts:
    - my.app.co.uk
    secretName: tls-secret

  rules:
  - host: my.app.co.uk
    http:
      paths:
      - path: /
        backend:
          serviceName: myappweb-service
          servicePort: 80

My nginx config map:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: nginx-ingress
    chart: nginx-ingress-0.28.3
    component: controller
    heritage: Tiller
    release: nginx-ingress
  name: nginx-ingress-controller
  namespace: default
data:
  proxy-buffer-size: "512k" -- this works and updates the nginx.conf

  location-snippet: |
    proxy_buffers 4 512k; -- this does not update the nginx.conf

  server-snippet: |       -- this does not update the nginx.conf
    location /messagehub {
      proxy_set_header Upgrade $http_upgrade;
      proxy_http_version 1.1;
      proxy_set_header X-Forwarded-Host $http_host;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Forwarded-For $remote_addr;
      proxy_set_header Host $host;
      proxy_set_header Connection "upgrade";
      proxy_cache_bypass $http_upgrade;
   }
like image 465
Declan McNulty Avatar asked Oct 12 '18 15:10

Declan McNulty


People also ask

What is nginx snippet?

Snippets allow you to insert raw NGINX config into different contexts of the NGINX configurations that the Ingress Controller generates. These should be used as a last-resort solution in cases where annotations and ConfigMap entries cannot help.

What is nginx ingress kubernetes IO configuration snippet?

The configuration-snippet is to add configs to locations. If you want to add a custom location to the server context, you should use the server-snippet instead: Using the annotation nginx.ingress.kubernetes.io/server-snippet it is possible to add custom configuration in the server configuration block.

What is proxy connect timeout?

proxy-connect-timeout : this defines the timeout for establishing a connection with a proxied server. The default value is 60 seconds, and the timeout typically cannot exceed 75 seconds. Check here for more information. proxy-send-timeout : this will set a timeout for transmitting a request to the proxied server.

What is nginx ingress kubernetes IO proxy buffer size?

This size can be configured by the parameter client_max_body_size and is set to 1m (1 Megabyte) by default. To configure this setting globally for all Ingress rules, the proxy-body-size value may be set in the NGINX ConfigMap.


2 Answers

It turns out that my problem was due to the content of the snippet that I was applying. Every time you run kubectl apply -f myconfigmap.yaml, a validation is run against the changes that you are trying to apply to the nginx.conf. When this validation fails, it fails silently and there is nothing to alert you to this in the terminal.

In fact, you still get the configmap/nginx-ingress-controller configured message.

For example, when I add this to the config map, it updates the nginx.conf as expected:

http-snippet: |
  sendfile on;

However, when I add this, nothing changes:

http-snippet: |
  sendfile on;
  tcp_nopush on;

The reason being that this has failed validation, but the only way to find that out is to look at the logs of the nginx ingress controller pod. In this instance I see:

Error: exit status 1
2018/10/16 07:45:49 [emerg] 470#470: "tcp_nopush" directive is duplicate in 
/tmp/nginx-cfg468835321:245
nginx: [emerg] "tcp_nopush" directive is duplicate in /tmp/nginx-cfg468835321:245
nginx: configuration file /tmp/nginx-cfg468835321 test failed

So I was duplicating a directive that already existed.

like image 26
Declan McNulty Avatar answered Sep 28 '22 23:09

Declan McNulty


If you'd like to modify your Kubernetes Ingress the annotation options are these:

  • nginx.ingress.kubernetes.io/configuration-snippet for an nginx location block snippet
  • nginx.ingress.kubernetes.io/server-snippet for a snippet in the nginx config service block

Looks like you are using nginx.org/location-snippets: for that case.

There's also a YAML invalid syntax on nginx config example and also you should use plurals as in server-snippets according to this example. There's a typo in the docs as of this writing. Opened this ticket to follow up.

It should be something like this:

  server-snippets: |
    location /messagehub {
      proxy_set_header Upgrade $http_upgrade;
      proxy_http_version 1.1;
      proxy_set_header X-Forwarded-Host $http_host;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Forwarded-For $remote_addr;
      proxy_set_header Host $host;
      proxy_set_header Connection "upgrade";
      proxy_cache_bypass $http_upgrade;
      }

instead of this:

  server-snippet: |
    location /messagehub {
      proxy_set_header Upgrade $http_upgrade;
      proxy_http_version 1.1;
      proxy_set_header X-Forwarded-Host $http_host;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Forwarded-For $remote_addr;
      proxy_set_header Host $host;
      proxy_set_header Connection "upgrade";
      proxy_cache_bypass $http_upgrade;
    }

Notice the indentation of the last curly brace.

like image 143
Rico Avatar answered Sep 28 '22 23:09

Rico