Logs not working on Elastic Beanstalk Tomcat environment

Question

We've been using Elastic Beanstalk to run our Java (8) applications on Tomcat in our company without any issues. Now we have decided to move on to Java 11. We set up our environments with CloudFormation as we did before for the Java 8 applications but now we are using this solution stack: '64bit Amazon Linux 2 v4.1.1 running Tomcat 8.5 Corretto 11' (also tried v4.1.2). Everything works fine but it looks like the request logs feature isn't working anymore in elastic beanstalk (Last 100 lines and full logs). I also added a keypair to the application server to check the logs on the ec2 server at /var/log/tomcat/ but we can't find the catalina.out file (only catalina files with a date like: catalina.2020-10-14.log). Those files contains only logs of a library we used but not the logs we send to the Standard system output. Another company we sometimes work with experiences the same issue on this solution stack.

Has anyone experienced the same issue on this solution stack and found a fix?

This is our CloudFormation configuration (YAML):

Application:
Type: AWS::ElasticBeanstalk::Application
Properties:
Description: Application backend
ApplicationVersion:
Type: AWS::ElasticBeanstalk::ApplicationVersion
Properties:
ApplicationName:
Ref: Application
Description: !Ref AppVersion
SourceBundle:
S3Bucket: !Ref BeanstalkSourceS3
S3Key: !Ref BuildFileName
AppEC2ServiceRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- elasticbeanstalk.amazonaws.com
Action:
- sts:AssumeRole
ManagedPolicyArns:
- "arn:aws:iam::aws:policy/service-role/AWSElasticBeanstalkEnhancedHealth"
- "arn:aws:iam::aws:policy/service-role/AWSElasticBeanstalkService"
AppEnvironment:
Type: AWS::ElasticBeanstalk::Environment
Properties:
ApplicationName:
Ref: Application
OptionSettings:
- Namespace: aws:autoscaling:launchconfiguration
OptionName: InstanceType
Value: !Ref EC2InstanceSize
- Namespace: aws:elasticbeanstalk:environment
OptionName: EnvironmentType
Value: SingleInstance
- Namespace: aws:autoscaling:launchconfiguration
OptionName: IamInstanceProfile
Value: !Ref AppInstanceProfile
- Namespace: aws:autoscaling:launchconfiguration
OptionName: EC2KeyName
Value: CompanyTestKey
- Namespace: aws:ec2:vpc
OptionName: VPCId
Value: !Ref Vpc
- Namespace: aws:ec2:vpc
OptionName: Subnets
Value: !Join [",",https://forums.aws.amazon.com/
- Namespace: aws:ec2:vpc
OptionName: AssociatePublicIpAddress
Value: true
- Namespace: aws:elasticbeanstalk:environment
OptionName: ServiceRole
Value: !Ref AppEC2ServiceRole
- Namespace: aws:elasticbeanstalk:cloudwatch:logs
OptionName: StreamLogs
Value: true
- Namespace: aws:elasticbeanstalk:cloudwatch:logs
OptionName: RetentionInDays
Value: 14
SolutionStackName: 64bit Amazon Linux 2 v4.1.1 running Tomcat 8.5 Corretto 11
VersionLabel:
Ref: ApplicationVersion
Tags:
- Key: group
Value: !Ref ResourceGroupTagValue

Answer 1

I've detailed in a new Medium blog how this all works for Tomcat, I hit similar issues, particularly zero-length files due to S3 Log Rotation.

Below is an excerpt that you might be able to use, the article explains how to determine the right folder/file(s) to stream.

Here I'm streaming three different files (it deals with the dates you mention automatically), review what's under the logs.conf 'content' section for each file that you need to stream.

Don't forget to also setup your policy and role that enables the Elastic Beanstalk EC2 instance to communicate with CloudWatch.

packages:
  yum:
    awslogs: []

option_settings:
  - namespace: aws:elasticbeanstalk:cloudwatch:logs
    option_name: StreamLogs
    value: true
  - namespace: aws:elasticbeanstalk:cloudwatch:logs
    option_name: DeleteOnTerminate
    value: false
  - namespace: aws:elasticbeanstalk:cloudwatch:logs
    option_name: RetentionInDays
    value: 90

files:
  "/etc/awslogs/awscli.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [plugins]
      cwlogs = cwlogs
      [default]
      region = `{"Ref":"AWS::Region"}`

  "/etc/awslogs/config/logs.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [/var/log/tomcat/localhost.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/tomcat/localhost.log"]]}`
      log_stream_name = {instance_id}
      file = /var/log/tomcat/localhost.*

      [/var/log/tomcat/catalina.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/tomcat/catalina.log"]]}`
      log_stream_name = {instance_id}
      file = /var/log/tomcat/catalina.*

      [/var/log/tomcat/localhost_access_log.txt]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/tomcat/access_log"]]}`
      log_stream_name = {instance_id}
      file = /var/log/tomcat/access_log.*

commands:
  "01":
    command: systemctl enable awslogsd.service
  "02":
    command: systemctl restart awslogsd