No logs appear on Cloudwatch log group for elastic beanstalk environment

Question

I have an elastic beanstalk environment, which is running a docker container that has a node js API. On the AWS Console, if I select my environment, then go to Configuration/Software I have the following:

• Log groups: /aws/elasticbeanstalk/my-environment
• Log streaming: Enabled
• Retention: 3 days
• Lifecycle: Keep after termination.

However, if I click on that log group on the Cloudwatch console, I have a Last Event Time of some weeks ago (which I believe corresponds to when the environment was created) and have no content on the logs.

Since this is a dockerized application, Logs for the server itself should be at /aws/elasticbeanstalk/my-environment/var/log/eb-docker/containers/eb-current-app/stdouterr.log. If I instead get the Logs directly from the instances by going once again to my EB environment, clicking "Logs" and then "Request last 100 Lines" the logging is happening correctly. I just can't see a thing when using CloudWatch.

Any help is gladly appreciated

Answer 1

I was able to get around this problem. So CloudWatch makes a hash based on the first line of your log file and the log stream key, and the problem is that my first line on the stdouterr.log file was actually an empty line!

After couple of days playing around and getting help from the good AWS support team, I first connected via SSH to my EC2 instance associated to the EB environment and you need to add the following line to the /etc/awslogs/config/beanstalklogs.conf file, right after the "file=/var/log/eb-docker/containers/eb-current-app/stdouterr.log" line:

file_fingerprint_lines=1-20

With these, you tell the AWS service that it should calculate the hash using lines 1 through 20 on the log file. You could change 20 for larger or smaller numbers depending on your logging content; however I don't know if there is an upper limit for the value.

After doing so, you need to restart the AWS Logs Service on the instance.

For this you would execute:

• sudo service awslogs stop
• sudo service awslogs start

or simpler:

sudo service awslogs restart

After these steps I started using my environment and the logging was now being properly streamed to the CloudWatch console! However this would not work if a new deployment is made, if the EC2 instance gets replaced or the auto scalable group spawns another.

To have a fix for this, it is possible to add log config via the .ebextensions directory, at the root of your application before deploying.

I added a file called logs.config to the newly created .ebextensions directory and placed the following content:

files:
  "/etc/awslogs/config/beanstalklogs.conf":
    mode: "000644"
    user: root
    group: root
    content: |
      [/var/log/eb-docker/containers/eb-current-app/stdouterr.log]
      log_group_name=/aws/elasticbeanstalk/EB-ENV-NAME/var/log/eb-docker/containers/eb-current-app/stdouterr.log
      log_stream_name={instance_id}
      file=/var/log/eb-docker/containers/eb-current-app/*stdouterr.log
      file_fingerprint_lines=1-20

commands:
  01_remove_eb_stream_config:
    command: 'rm /etc/awslogs/config/beanstalklogs.conf.bak'
  02_restart_log_agent:
    command: 'service awslogs restart'

Changing of course EB-ENV-NAME by my environment name on EB.

Hope it can help someone else!

Answer 2

For 64 bit Amazon Linux 2 the setup is slightly different.

For the delivery of log the AWS CloudWatch Agent is installed in /opt/aws/amazon-cloudwatch-agent and the Elastic Beanstalk configuration is in /opt/aws/amazon-cloudwatch-agent/etc/beanstalk.json. It is set to log the output of the container assuming there's a file called stdouterr.log, here's a snippet of the config:

{
  "file_path": "/var/log/eb-docker/containers/eb-current-app/stdouterr.log",
  "log_group_name": "/aws/elasticbeanstalk/EB-ENV-NAME/var/log/eb-docker/containers/eb-current-app/stdouterr.log",
  "log_stream_name": "{instance_id}"
}

However when I look for the file_path it doesn't exist, instead I have a file path that encodes the current docker container ID /var/log/eb-docker/containers/eb-current-app/eb-e4e26c0bc464-stdouterr.log.

This logfile is created by a script /opt/elasticbeanstalk/config/private/eb-docker-log-start that is started by the eb-docker-log service, the default contents of this file are:

EB_CONFIG_DOCKER_CURRENT_APP=`cat /opt/elasticbeanstalk/deployment/.aws_beanstalk.current-container-id | cut -c 1-12`
mkdir -p /var/log/eb-docker/containers/eb-current-app/
docker logs -f $EB_CONFIG_DOCKER_CURRENT_APP >> /var/log/eb-docker/containers/eb-current-app/eb-$EB_CONFIG_DOCKER_CURRENT_APP-stdouterr.log 2>&1

To temporarily fix the logging you can manually run (replacing the docker ID) and then logs will start to appear in CloudWatch:

ln -sf /var/log/eb-docker/containers/eb-current-app/eb-e4e26c0bc464-stdouterr.log /var/log/eb-docker/containers/eb-current-app/stdouterr.log

To make this permanant I added an .ebextension to fix the eb-docker-log service so it re-makes this link so create a file in your source code in .ebextensions called fix-cloudwatch-logging.config and set it's contents to:

files:
  "/opt/elasticbeanstalk/config/private/eb-docker-log-start" :
    mode: "000755"
    owner: root
    group: root
    content: |
      EB_CONFIG_DOCKER_CURRENT_APP=`cat /opt/elasticbeanstalk/deployment/.aws_beanstalk.current-container-id | cut -c 1-12`
      mkdir -p /var/log/eb-docker/containers/eb-current-app/
      ln -sf /var/log/eb-docker/containers/eb-current-app/eb-$EB_CONFIG_DOCKER_CURRENT_APP-stdouterr.log /var/log/eb-docker/containers/eb-current-app/stdouterr.log
      docker logs -f $EB_CONFIG_DOCKER_CURRENT_APP >> /var/log/eb-docker/containers/eb-current-app/eb-$EB_CONFIG_DOCKER_CURRENT_APP-stdouterr.log 2>&1
commands:
  fix_logging:
    command: systemctl restart eb-docker-log.service
    cwd: /home/ec2-user
    test: "[ ! -L /var/log/eb-docker/containers/eb-current-app/stdouterr.log ] && systemctl is-active --quiet eb-docker-log"