Menu
Has anyone built, or does anyone know how to build, a logging framework that deposits the log files on Amazon S3?
We're building a SaaS app, and, naturally, we plan to have a zillion servers and customers. I'd like to simplify the entire logging structure. At the moment we use SLF4j for logging and Logback as the actual logging implementation.
I'm thinking that we could drop in another implementation that would accumulate log messages in memory, compress them, and then upload them to S3. If the S3 connection were down or slow, the implementation would queue the files up on disk. Kind of like Facebook's Scribe.
My guess is that it would take me four or five days to write and test this. If there's another implementation out there, I'd love to know about it.
609
Sign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/ . In the navigation pane on the left side of the console, choose access points. On the access points page, under access points, select the AWS Region that contains the access points you want to list.
Amazon S3 Logs help you keep track of data access and maintain a detailed record of each request. These include resources specified in the request, request type, along with the date and time the request was processed. Once you enable the logging process, these are written to an Amazon S3 bucket.
You can record the actions that are taken by users, roles, or AWS services on Amazon S3 resources and maintain log records for auditing and compliance purposes. To do this, you can use server access logging, AWS CloudTrail logging, or a combination of both.
You can access an S3 bucket privately without authentication when you access the bucket from an Amazon Virtual Private Cloud (Amazon VPC). However, make sure that the VPC endpoint used points to Amazon S3.
There is a plugin for fluentd that stores files to s3. (Fluentd is a nice "log file collector")
Read more about it here: https://docs.fluentd.org/output/s3
If the s3 connection is down or slow it will buffer the output for you.
134
You can try to write a custom appender for logback or log4j2 and use this appender in respective configuration.
This way, you don't have to write an entire logging framework, but only the part you need and use the rest from a working framework.
There are also a few of them on github. For instance: shuwada/logback-s3
41
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
