.Net Core behind NGINX returns 502 Bad Gateway after authentication by IdentityServer4

Question

Having two applications auth and store and authenticating using IdentityServer4 and both are behind NGINX.

The store application successfully authenticates but after coming back from the auth application we get 502 Bad Gateway from NGINX.

Any idea what is going wrong here?

Auth app log:

info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
      Request finished in 117.7292ms 200 text/html; charset=UTF-8
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
      Request starting HTTP/1.0 POST http://auth.example.com/connect/token application/x-www-form-urlencoded 279
info: IdentityServer4.Hosting.IdentityServerMiddleware[0]
      Invoking IdentityServer endpoint: IdentityServer4.Endpoints.TokenEndpoint for /connect/token
info: IdentityServer4.Validation.TokenRequestValidator[0]
      Token request validation success
      {
        "ClientId": "ExampleStore",
        "ClientName": "Example Web Store",
        "GrantType": "authorization_code",
        "AuthorizationCode": "6fab1723...",
        "Raw": {
          "client_id": "ExampleStore",
          "client_secret": "***REDACTED***",
          "code": "6fab1723...",
          "grant_type": "authorization_code",
          "redirect_uri": "https://store.example.com/signin-oidc"
        }
      }
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
      Request finished in 182.8022ms 200 application/json; charset=UTF-8
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
      Request starting HTTP/1.0 GET http://auth.example.com/connect/userinfo
info: IdentityServer4.Hosting.IdentityServerMiddleware[0]
      Invoking IdentityServer endpoint: IdentityServer4.Endpoints.UserInfoEndpoint for /connect/userinfo
info: IdentityServer4.ResponseHandling.UserInfoResponseGenerator[0]
      Profile service returned to the following claim types: sub preferred_username name
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
      Request finished in 57.1394ms 200 application/json; charset=UTF-8

Store app log:

info: Microsoft.AspNetCore.Authorization.DefaultAuthorizationService[2]
      Authorization failed for user: (null).
info: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[3]
      Authorization failed for the request at filter 'Microsoft.AspNetCore.Mvc.Authorization.AuthorizeFilter'.
info: Microsoft.AspNetCore.Mvc.ChallengeResult[1]
      Executing ChallengeResult with authentication schemes ().
info: Microsoft.AspNetCore.Authentication.OpenIdConnect.OpenIdConnectHandler[12]
      AuthenticationScheme: oidc was challenged.
info: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[2]
      Executed action Nihonto.Web.Store.Controllers.UserController.Login (Nihonto.Web.Store) in 8.1968ms
info: Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware[27]
      The response could not be cached for this request.
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
      Request finished in 11.2816ms 302
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
      Request starting HTTP/1.0 POST http://store.example.com/signin-oidc application/x-www-form-urlencoded 1485
info: Microsoft.AspNetCore.Authentication.Cookies.CookieAuthenticationHandler[10]
      AuthenticationScheme: ExampleCookie signed in.
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
      Request finished in 301.361ms 302

More information of this issue can be found here : https://github.com/IdentityServer/IdentityServer4/issues/2101

Answer 1

The problem has been solved. It seems that NGINX doesn't allow a large header content. From this help https://medium.com/@mshanak/solve-nginx-error-signin-oidc-502-bad-gateway-dotnet-core-and-identity-serve-bc27920b42d5 , we have set these properties:

nginx.conf

http{
...
proxy_buffer_size   128k;
proxy_buffers   4 256k;
proxy_busy_buffers_size   256k;
large_client_header_buffers 4 16k;
...
}

default.conf

location /{
    ...
    fastcgi_buffers 16 16k;
    fastcgi_buffer_size 32k;
    ...
}

Wonder if there is any way to configure IdentityServer to send much smaller header content!

Answer 2

It is also possible to configure this with annotations:

  annotations:
    kubernetes.io/ingress.class: "nginx"    
    nginx.ingress.kubernetes.io/proxy-buffering: "on"
    nginx.ingress.kubernetes.io/proxy-buffer-size: "128k"
    nginx.ingress.kubernetes.io/proxy-buffers-number: "4"

so you can add them to your existing ingress.yaml, for example like that:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-production
  namespace: ingress-nginx
  annotations:
    kubernetes.io/ingress.class: "nginx"    
    nginx.ingress.kubernetes.io/proxy-buffering: "on"
    nginx.ingress.kubernetes.io/proxy-buffer-size: "128k"
    nginx.ingress.kubernetes.io/proxy-buffers-number: "4"
    certmanager.k8s.io/issuer: "letsencrypt-production"
spec:
  tls:
  - hosts:
    - example.com
    secretName: example-tls
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: example-app
          servicePort: 80

Answer 3

The answers saved my day. Just want to mention, that additional thought should be put into configuration values. Tuning proxy_buffer_size in NGINX

For me following values were sufficient:

nginx.ingress.kubernetes.io/proxy-buffer-size: 8k
nginx.ingress.kubernetes.io/proxy-busy-buffers-size: 16k