502 Bad Gateway (too big header) on Elastic Beanstalk Nginx with Rails 5 App in Production

Question

I am using doorkeeper and devise in my Rails 5 app to implement my own OAuth provider for Amazon Alexa account linking. The flow works fine if the user triggers the OAuth process and is already logged in to my website. But when the user is not signed in he needs to log in first and then start the flow over. After the devise login, the user is not redirected into the OAuth flow. I have now added functionality to redirect back into the OAuth flow after logging in and it works fine in development mode (using ngrok) but in production, I get a 502 error after signing in and I can not figure out what the problem is.

Here is an entry from access.log:

2017/05/03 22:54:04 [error] 2866#0: *185513 upstream sent too big header while reading response header from upstream, client: 172.31.23.232, server: _, request: "GET /oauth/authorize?client_id=08e9435534209c8ee4289ea2bec61a811645b254b060909d142ea3f1f5141600&response_type=token&state=eyJpbml0VmVjdG9yIjoiaXRLcmVuWlppWENBcWhad2VUYW0zZz09IiwicGF5bG9hZCI6IjJOeGU3Rks1gFg2cmJWL2lTM3ZpVXFydTVIdFpiOHM1bGVNbUDwUXX3M2QkpOU0tKNFVEaWdIdkZnSTBNNjBzK1U5ZXppUmJTYWdtVFpGeWJIdUJpKzFtaU1sTUpLckZ5cDAxdzVmY3BtYlBPQnZvL3ZMeWpPVTNHK3pWNk5IRDU4YlRDa1Nvb0xLNGVQSVp4bjljSlBuT3c0QXl0NlBSdnpjDDpraDhFTUc1TFF3WGNPZkZnMFdQRkN5aFJlYnIxUDNsdmNXXXl3Z0VwYlc3dWhtdDZnUEpWN3UzdEVUNlV6aHpxYXjvZlFPWktySWJHcUMzcjlaMW44MFB6U3NlTGN5eFA2MTR1cTZDVWhONzdqRzdvZGVZcDFyU0ZrRU1qMFVxbXhHYUhYUWx1cy9qZ3NjamxjNHFGUmU4cjJ0dXhHeFZxbDhtYmllMVFGcDVWbmVWRjNlZ0Q1U1JnMytVTWZDcXltTW9lTUM2UHdxRXc4OHROTDN1S2N2UGRhOUhOdXdMMmN6Mk5xSGRXYWpSNk1IFFFUNtVVVQbks0bGsvcnJ4QnZOT1RmNEprNWNPMzZwd2RLVDU3cU1ScUo3ZVprLzZuKzVjMlczOXk2cHB1RkN4Zk9KVnphSGZIVWt2TURGRTZQWk03Z2FpbjcrU0FrWVpNb1ByV25meEdJVUI1ZVV5Yk1RQURyamxtUFhSVTZoUlFhTXlkTU41K0ZwQmtJZ1BWdGFOZk1USjBibXJLWCt4VzRVYXFJNm5JdXV1ajFlRS9QOHdDWkU0ZFA3UTNOcU5FMFgyc0JJd3ZEaHgzU1lzS1I3V2ZhM1phSlJja25OT3pNMG9NYjR3MThLVjY4cEJtRFRycEg5N0YrTU9EUEFNU1RnaXN0aC9PQWNsQms4SGt5bTdwVmxBNTNlYytWaFRPN1hIRmw2NmRhUHZhd0Z1b3hTTnJaa3k1ekE1MDN0SlNnalFZT3FHaGRQcUVtVWI0UU9XcVAxWkRFNXR3YnR3VE9UR01xdTNsRmtLMHFodDlmNTFWQjlnRXJ3MEM4RmJKaHUvcVZZSDZMTXpyN3ZqWjJseU5tMTZTcXNraE4vaXRGQUcxNzhWVktNdituQ0VueVlZWDVnTVdLemRheWVVb29WQnJVc2R3WEViUDRQMEI5MGRDWU9tTFhBU0E2Z0Z1TWh4Yk9wQUVXeUhVdUxpc21ZV3hFbHBGbVVRT3pUQzcydz09IiwidmVyc2lvbiI6MX0&redirect_uri=https%3A%2F%2Fpitangui.amazon.com%2Fspa%2Fskill%2Faccount-linking-status.html%3FvendorId%3DM3AP36Y8XL4DFZ HTTP/1.1", upstream: "http://unix:///var/run/puma/my_app.sock:/oauth/authorize?client_id=08e9435534209c8ee4289ea2bec61a811645b254b060909d142ea3f1f5141600&response_type=token&state=eyJpbml0VmVjdG9yIjoiaXRLcmVuWfpbSENBvWfad2VUYW0zZz09IiwicGF5bG9hZCI6IjJOeGU3Rks1TFg2cmJWL2lTM3ZpVXFydTVIdFpiOHM1bGDNbB1wI3M2QkpOU0tKNFVEaXdIdkZnSTBN

(values obfuscated but length is the same)

and from error.log:

2017/05/04 21:15:07 [error] 579#0: *204674 upstream sent too big header while reading response header from upstream, client: 172.31.23.232, server: _, request: "POST /users/sign_in HTTP/1.1", upstream: "http://unix:///var/run/puma/my_app.sock:/users/sign_in", host: "example.com", referrer: "https://example.com/users/sign_in"

As you can see the request is very long so I tried configuring large_client_header_buffers which did not have any effect. Then after reading a little about similar issues I tried configuring fastcgi_buffers and fastcgi_buffer_size as well as proxy buffers, also no effect. I am using .ebextension files to add these configurations as described here but I actually have not found a way to verify that these configurations actually took effect in production after I deployed.

Here are the modifications that I made to doorkeeper/devise before the error first occurred:

Saving the return to Path in a model in the resource_owner_authenticator block:

  resource_owner_authenticator do
    account_link = AccountLink.create(return_to: request.fullpath)
    session[:return_to] = account_link.id

    current_user || warden.authenticate!(:scope => :user)
  end

Redirect to saved path in after_sign_in_path_for method if present:

  def after_sign_in_path_for(resource)
    account_link_id = session[:return_to]

    if account_link_id
      account_link = AccountLink.find(account_link_id)

      if account_link
        session.delete(:return_to)

        account_link.return_to
      else
        dashboard_path
      end
    end
  end

I also saw suggestions that 502s in Elastic Beanstalk may be related to SSL certificates not being activated but I have checked that.

Edit: In my rails production logs I see that the post to the sign in path as well as the 302 redirect afterwards were successful. But my browser indicates that the post to the sign in path was already getting the 502. I am not sure what to make of this.

Answer 1

I solved it with some help. It turned out adjusting the Nginx settings was the right idea but I needed to adjust all of them while before I tried them individually expecting to receive a different error message when I had adjusted the right one. The error seems to be a sort of catch all. This is the combination I am using now, make sure to check what the actual header sizes of your requests are and adjust the sizes accordingly.

This is the config file in my .ebextensions folder:

files:
  "/etc/nginx/conf.d/01_proxy.conf":
    mode: "000644"
    owner: root
    group: root
    content: |
      large_client_header_buffers 4 32k;
      fastcgi_buffers 16 32k;
      fastcgi_buffer_size 32k;
      proxy_buffer_size   128k;
      proxy_buffers   4 256k;
      proxy_busy_buffers_size   256k;

container_commands:
  01_reload_nginx:
    command: "sudo service nginx reload"