Enhanced Health Overview not reporting properly for Multi-container Docker configuration in AWS Elastic Beanstalk

Question

I uploaded a project having multi-containers docker platform with two containers say xyz and abc in aws elastic-beanstalk. xyz contains tomcat server in it. I have following configuration in my project for Dockerrunner.aws.json file.

{
  "AWSEBDockerrunVersion": 2,
  "containerDefinitions": [
    {
      "name": "xyz",
      "image": "<PLACEHOLDER_REPLACED_BY_CICD_TOOLS>",
      "essential": true,
      "memory": 2048,
      "links": [
        "abc"
      ],
      "environment": [
        {
          "name": "ENVIRONMENT",
          "value": "QA"
        },
        {
          "name": "LOG_HOME",
          "value": "/usr/local/tomcat/logs"
        },
        .
        .
        .
      ],
      "mountPoints": [
        {
          "sourceVolume": "awseb-logs-xyz",
          "containerPath": "/usr/local/tomcat/logs"
        }
      ],
      .
      .
    },
    {
      "name": "abc",
      "image": "image123",
      "essential": true,
      .
      .
      .
    }
  ]
}

But, I am not able to view data in health section of elastic-beanstalk.

What I did so far to resolve this issue:

• I read (https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/health-enhanced-serverlogs.html) and got to know that elastic beanstalk has special logging format for multi-container health page to work.
• For testing propose, I manually created an error log-file in the same format by accessing ec2 instance. The file I created in ec2 instance host at location /var/log/containers/xyz (where health-agent read logs) was also mapped properly to tomcat's log file's location (i.e. /usr/local/tomcat/logs) in xyz docker container.

But, I still could not see changes in enhanced health overview section.

Answer 1

From this AWS support site:

From the Elastic Beanstalk console, verify that enhanced health reporting is enabled:

1. Choose Configuration, and then on the Health panel under Web Tier, choose the edit gear.
2. Under Health Reporting, ensure the System type is set to Enhanced.

64-bit Amazon Linux 2016.xx vx.x.x running Node.js platform:

Ensure that the correct proxy server is configured:

1. Choose Configuration, and then on the Software Configuration panel under Web Tier, choose the edit gear.

2. In Container Options, ensure you have a Proxy server selected. If Proxy server is set to none, the application log file is not generated under /var/log/nginx/healthd/ and health reporting does not generate data to display.

You can also modify the Node.js logs and location to be compatible with enhanced health log format, then review the healthd configuration file /etc/healthd/config.yaml.

64-bit Amazon Linux 2016.xx vx.x.x running Multicontainer Docker 2.xx.x:

This platform doesn’t come with a proxy server, so you need to ensure that logs are produced in the correct format from their containers and configure healthd to read them. To use enhanced health monitoring in Multicontainer Docker environments, you need to configure healthd to consume these logs.

To provide logs to the health agent, ensure the following:

• Logs are in the correct format

• Logs are written to /var/log/nginx/healthd/

• Log names use the format: application.log.$year-$month-$day-$hour

• Logs are rotated once per hour

• Logs are not truncated

Note: With the Node.js platform, if you disable the proxy, the logs are not created under /var/logs/nginx/healthd/. You must either re-enable the proxy or configure your Node.js application to produce logs under /var/logs/nginx/healthd/

This sample Docker-multicontainer-v2.zip code shows how to manage ebextensions where the healthd configuration is set to read another directory. [...]

I think this part might be able to help you:

If you are unable to see information for a server in the Enhanced Health Overview, check the healthd service status on the instance and ensure that it’s running. If it is not running, restart the service.

This sample code shows how to check the healthd service status:

$ ps aux | grep healthd

This sample code shows how to restart the healthd service:

[ec2-user@ip-172-31-39-182 ~]$ sudo initctl restart healthd