Storing log file in AWS S3 from Spring boot application

Question

I am building a Spring-boot application where in all the logging is stored in a specified path in log4j.properties.

I am hosting this application on AWS Beanstalk.

Once I host the application on the AWS, the specified path for the log file will become invalid.

How can I resolve this issue where in the log file should also get stored in a different path in cloud, say Amazon S3, but not on server as the log file takes considerable amount of size.

log4j.appender.file.File=/my_log.log

How can I change the above line to store the "my_log.log" in AWS S3?

Answer 1

    import com.amazonaws.services.s3.AmazonS3;
    import com.amazonaws.services.s3.model.PutObjectRequest;
    import org.springframework.core.env.Environment;
    import org.springframework.scheduling.annotation.EnableScheduling;
    import org.springframework.scheduling.annotation.Scheduled;


@Autowired
    private AmazonS3 s3Client;
@Scheduled(cron = "0 5 1 * * *")
public void moveLogsFromEC2ToS3() {
    try {
        File logsDir = new File(env.getProperty("AWS_EC2_LOG_PATH"));
        for (File logFile : logsDir.listFiles()) {
            String fileName = logFile.getName();
            if (fileName.endsWith(".log")) {
                s3Client.putObject(new PutObjectRequest(env.getProperty("AWS_S3_LOGS_BUCKET_NAME"), fileName, logFile));
            }
        }
    } catch (Exception e) {
        logger.error("Error in moving log files! : {}", e);
    }
}

Answer 2

Although user has mentioned log4j, in case someone is using logback, they could use this library. https://github.com/link-nv/logback-s3-rolling-policy

logback-s3-rolling-policy automatically uploads rolled log files to S3.

There are 2 rolling policies which can be used:

- S3FixedWindowRollingPolicy
- S3TimeBasedRollingPolicy

User would need to use these Rolling policies in their RollingFileAppender configuraiton

Example:

<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
  <file>logs/myapp.log</file>
  <encoder>
    <pattern>[%d] %-8relative %22c{0} [%-5level] %msg%xEx{3}%n</pattern>
  </encoder>
  <rollingPolicy class="ch.qos.logback.core.rolling.S3FixedWindowRollingPolicy">
    <fileNamePattern>logs/myapp.%i.log.gz</fileNamePattern>
    <awsAccessKey>ACCESS_KEY</awsAccessKey>
    <awsSecretKey>SECRET_KEY</awsSecretKey>
    <s3BucketName>myapp-logging</s3BucketName>
    <s3FolderName>logs/%d{yyyy/MM/dd}</s3FolderName>
    <rolloverOnExit>true</rolloverOnExit>
    <shutdownHookType>SERVLET_CONTEXT</shutdownHookType>
    <prefixTimestamp>true</prefixTimestamp>
    <prefixIdentifier>true</prefixIdentifier>
  </rollingPolicy>
  <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
    <maxFileSize>10MB</maxFileSize>
  </triggeringPolicy>
</appender>