AWS EB + Nginx, update access.log format or create new log

Question

I'm running an app on AWS' Elastic Beanstalk using the configuration Node.js running on 64bit Amazon Linux/4.5.0, with Nginx.

I would like to add the request header "X-My-Header" as a field to the access.log. Barring that, I would take creating a new log file using the compound default nginx logs + my header. I've found several similar questions specifically about logging with nginx, but the EB aspect throws an extra curveball with how the nginx configs are updated through an /.ebextensions config file.

I've accomplished creating a log file, but it isn't getting populated with anything. I also tried just updating the access.log file, but that doesn't seem to have taken, either. I saw other people adding headers would use the format "$http_", and it seems like an http request header of "X-Header-Example" gets formatted to "$http_header_example" (see "$http_user_agent" in the nginx compound default), though not wanting to waste time with the assumption, note that I added both "$http_x-my-header" and "$http_x_my_header".

Attempt 1: Update existing access.log format

files:
  /etc/nginx/conf.d/01_proxy.conf:
      owner: root
      group: root
      content: |
          log_format my_log_format '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" - "$http_x_my_header" - "$http_x-my-header"';
          access_log /var/log/nginx/access.log my_log_format;

Result: access.log does not include any additional fields. It doesn't even have empty ""s, or the -.

Attempt 2: Create a new log file

files:
  /etc/nginx/conf.d/01_proxy.conf:
      owner: root
      group: root
      content: |
          log_format my_log_format '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" - "$http_x_my_header" - "$http_x-my-header"';
          access_log /var/log/nginx/new_log.log my_log_format;

Result: new_log.log now appears in var/log/nginx when I export logs from the EB dashbaord. However, it's completely empty.

I read some other similar questions mentioning deleting files and restarting the server sometimes helps. I tried restarting the application and even completely rebuilding the environment through the EB dashboard, and neither led to different results.

I largely based my solution on this medium article, section 2.1. However, when I tried adding the container_command to my .config file, my entire environment stopped working. I had to revert to a different deployment, and then rebuild the environment to get it running again.

Any tips?

My goal is to associate this request header with the requests coming in. Ideally I could update the existing default access.log. I will settle for a separate file. Or, if you have any other suggestions as to how I may be able to get access to this info, I'm all ears! Thanks.

Edit A new attempt:

Here it shows that you can completely replace the default nginx.config, so I tried removing my other file and instead copy/pasting the default from the medium article from before into a /.ebextensions/nginx/nginx.config file, except adding my changes there. I updated log_format main to include my "$http_x_my_header" values.

Unfortunately, the deployment failed with this message:

The configuration file .ebextensions/nginx/nginx.config in application version contains invalid YAML or JSON. YAML exception: Invalid Yaml: expected '', but found Scalar in "", line 7, column 1: include /usr/share/nginx/modules ... ^ , JSON exception: Invalid JSON: Unexpected character (u) at position 0.. Update the configuration file.

The offending line is include /usr/share/nginx/modules, which exists and works fine in the default that medium article provided.

I was hoping this would be a dirty fix that I could at least get some results from, but alas, it seems to have another roadblock.

Answer 1

I've answered this question through my own answer in a similar question:

AWS EB + nginx: Update access.log format to obfuscate sensitive get request parameters

The short of it: For Node AWS EB environments, the server directive of the nginx config exists inside an auto generated 00_elastic_beanstalk_proxy.conf file. Within here, they call access_log /var/log/nginx/access.log main, so adding a ebextension config trying to change access_log gets overridden.

My solution was twofold: override the main log_format by uploading a custom nginx.conf based on the default (AWS says you can do this, but recommends you by pulling the one created by default, and re-checking it when you update the version of the environment's image), and I also had to do the same with the auto generated file to perform some logic that sets the new variable I wanted to log.

For more details, see the answer linked above, which has more information on the process.

Answer 2

My solution is to override the nginx.conf.

The AWS doc for nodejs platform is to delete the existing /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf and replace with your own config. I have tested it works as well. For Docker platform, you need to delete /sites-enabled/elasticbeanstalk-nginx-docker-proxy.conf.

This following code is for docker platform only. Version: Docker running on 64bit Amazon Linux/2.16.7

You should find the default code of nginx.conf in your platform.

Add .ebextensions/nginx/00-my-proxy.config to your build folder

files:
  /etc/nginx/nginx.conf:
    mode: "000644"
    owner: root
    group: root
    content: |
        # Elastic Beanstalk Nginx Configuration File

        user  nginx;
        worker_processes  auto;

        error_log  /var/log/nginx/error.log;

        pid /var/run/nginx.pid;

        events {
            worker_connections  1024;
        }

        http {
          include /etc/nginx/mime.types;
          default_type application/octet-stream;

          access_log /var/log/nginx/access.log;

          log_format healthd '$msec"$uri"$status"$request_time"$upstream_response_time"$http_x_forwarded_for';

          upstream docker {
              server 172.17.0.2:3000;
              keepalive 256;
          }
          log_format timed_combined '"$http_x_forwarded_for"'
                      '$remote_addr - $remote_user [$time_local] '
                      '"$request" $status $body_bytes_sent '
                      '"$http_referer" "$http_user_agent" '
                      '$request_time $upstream_response_time $pipe';


          map $http_upgrade $connection_upgrade {
                  default        "upgrade";
                  ""            "";
          }

          server {
              listen 80;

                gzip on;
              gzip_comp_level 4;
              gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

                if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2})") {
                    set $year $1;
                    set $month $2;
                    set $day $3;
                    set $hour $4;
                }
                access_log /var/log/nginx/healthd/application.log.$year-$month-$day-$hour healthd;

                access_log /var/log/nginx/access.log timed_combined;

                location / {
                    proxy_pass            http://docker;
                    proxy_http_version    1.1;

                    proxy_set_header    Connection            $connection_upgrade;
                    proxy_set_header    Upgrade                $http_upgrade;
                    proxy_set_header    Host                $host;
                    proxy_set_header    X-Real-IP            $remote_addr;
                    proxy_set_header    X-Forwarded-For        $proxy_add_x_forwarded_for;
                }
          }
        }

In EB docker platform, the server block in Nginx config is in another file /etc/nginx/sites-enabled/elasticbeanstalk-nginx-docker-proxy.conf

server {
    listen 80;
     access_log    /var/log/nginx/access.log;
}

The entry config nginx.conf is

    include       /etc/nginx/conf.d/*.conf;
    include       /etc/nginx/sites-enabled/*;

So adding this customized config will only lead to this result - empty log file.

files:
  /etc/nginx/conf.d/01_proxy.conf:
      owner: root
      group: root
      content: |
          log_format my_log_format '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" - "$http_x_my_header" - "$http_x-my-header"';
          access_log /var/log/nginx/new_log.log my_log_format;

   log_format ...
   access_log ...
   server {
       ...
   }

I have wrote the details in my github.