Nginx client_max_body_size not working in Docker container on AWS Elastic Beanstalk

Question

I'm having a problem where nginx seems to be ignoring (or overriding) my upped client_max_body_size directive in a Ubuntu Docker container on AWS Elastic Beanstalk. This is preventing users from uploading files any larger than the nginx default of 1MB.

I have used the client_max_body_size 10M; in http, server,and location blocks to no avail, I am still seeing "client intended to send too large body" errors in the nginx logs. I have successfully used these settings on an AWS EC2 Ubuntu instance, but since using the same setup in a Docker container I am having this problem. I've also tried using an ebextension as outlined here Increasing client_max_body_size in Nginx conf on AWS Elastic Beanstalk

The app itself is CFML (Railo) running in a Tomcat container.

Here are the relevant nginx files:

The full unabridged files are here https://github.com/chapmandu/docker-railo

Thanks in advance.

nginx error.log

2014/12/02 03:02:05 [error] 32116#0: *142 client intended to send too large body: 1290803 bytes, client: 172.31.19.39, server: , request: "POST /listings/35602/images/create HTTP/1.1", host: "staging.svr.com.au", referrer: "http://staging.svr.com.au/listings/35602/images/new"

nginx.conf

daemon off;

worker_processes  1;

events {
    worker_connections  1024;
}

http {
    client_max_body_size 10M;
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    include /etc/nginx/sites-enabled/default;
}

default

server
{
    listen       80;
    server_name  localhost;

    client_max_body_size 10M;

    # don't rewrite for static files
    location ~* ^.+\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js|flv|swf|html|htm|map|ttf|woff)$
    {
        root   /var/www;
    }

    location /
    {
        root   /var/www;
        index  index.cfm;
        client_max_body_size 10M;
        include proxy_params;
    }
}

proxy_params

proxy_redirect off;

# # If you want your server to identify itself only as Tomcat you can pass
# # the Tomcat setting to Nginx telling Nginx not to change it
#proxy_pass_header Server;

# Point Nginx to Tomcat
proxy_pass  http://localhost:8080;

# Send appropriate headers through
# Forward the real ip to Tomcat (and Railo)

proxy_buffers 16 16k;
proxy_buffer_size 32k;

# prevent regular 504 Gateway Time-out message
proxy_connect_timeout       600;
proxy_send_timeout          600;
proxy_read_timeout          600;
send_timeout                600;

# pass headers through
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Query-String $request_uri;
proxy_set_header X-Host $host;
proxy_set_header X-Remote-Addr $remote_addr;
proxy_set_header X-Request-Filename $request_filename;
proxy_set_header X-Request-URI $request_uri;
proxy_set_header X-Server-Name $server_name;
proxy_set_header X-Server-Port $server_port;
proxy_set_header X-Server-Protocol $server_protocol;

proxy_intercept_errors on;

# apparently this is how to disable cache?
expires     -1;

Answer 1

It turns out that AWS uses nginx to proxy connections to the docker container, it's necessary to update the AWS nginx configuration in order to set client_max_body_size. Note that I am using 64bit Amazon Linux 2014.09 v1.0.9 running Docker 1.2.0.

I needed to create .ebextensions/01-client-max-body.config with the contents below.

The important line is "server 127.0.0.1:EB_CONFIG_NGINX_UPSTREAM_PORT;"

files:
  "/tmp/custom-nginx.conf":
    mode: "00644"
    owner: "root"
    group: "root"
    content: |
      upstream docker {
        server 127.0.0.1:EB_CONFIG_NGINX_UPSTREAM_PORT;
        keepalive 256;
      }

      server {
        listen EB_CONFIG_HTTP_PORT;
        client_max_body_size 10M;

        location / {
          proxy_pass      http://docker;
          proxy_http_version      1.1;
          proxy_set_header Host                $host;
          proxy_set_header X-Real-IP           $remote_addr;
          proxy_set_header X-Forwarded-For     $proxy_add_x_forwarded_for;
        }
      }

container_commands:
  01_remove_orig_config:
    command: 'sed -i ''/EOF/,/EOF/d'' /opt/elasticbeanstalk/hooks/appdeploy/enact/00flip.sh '
  02_add_new_conf:
    command: 'sed -i ''/# set up nginx/a sed -i s\/EB_CONFIG_NGINX_UPSTREAM_PORT\/$EB_CONFIG_NGINX_UPSTREAM_PORT\/ \/tmp\/custom-nginx.conf \nsed -i s\/EB_CONFIG_HTTP_PORT\/$EB_CONFIG_HTTP_PORT\/ \/tmp\/custom-nginx.conf \ncat \/tmp\/custom-nginx.conf > \/etc\/nginx\/sites-available\/elasticbeanstalk-nginx-docker.conf'' /opt/elasticbeanstalk/hooks/appdeploy/enact/00flip.sh'
    test: 'test ! $(grep custom-nginx.conf /opt/elasticbeanstalk/hooks/appdeploy/enact/00flip.sh)'

Answer provided by super helpful AWS support..

Answer 2

The accepted answer didn't work for me, but I managed to figure it out.

Previously I used a prebuilt docker image (hosted in my own registry), and specified access and url in the dockerrun.aws.json file, and then had a config file at .elasticbeanstalk/config.json with the following:

deploy:
  artifact: Dockerrun.aws.json

This uploaded only the dockerrun file as an artifact.

In order to set the client_max_body_size via ebextentions, you must switch to either uploading the entire folder via eb deploy or specify a zip file artifact containing your .ebextentions folder

I then had success with a config file at .ebextentions/size.config containing:

files:
  /etc/nginx/conf.d/proxy.conf:
    content: |
      client_max_body_size 50M;
container_commands:
   01_reload_nginx:
     command: "service nginx reload"

I arrived at this answer thanks to https://stackoverflow.com/a/23587371/2653503, https://stackoverflow.com/a/39829789/2653503, and a few others that I can't find right now.

Answer 3

A few stabs in the dark ... I notice you have 2 occurrences of the client_max_body_size parameter in your example, and that your github configuration is different. If there is more than one occurrence of a directive in a config file or include, nginx will take the value of the directive defined last.

Another possibility to consider is that files uploaded via a web form will often have a larger file size than that shown in the file system. Have you tried a extra large setting, such as

client_max_body_size 500m;

Reference: Mastering Nginx