Logging to Amazon S3

Question

Has anyone built, or does anyone know how to build, a logging framework that deposits the log files on Amazon S3?

We're building a SaaS app, and, naturally, we plan to have a zillion servers and customers. I'd like to simplify the entire logging structure. At the moment we use SLF4j for logging and Logback as the actual logging implementation.

I'm thinking that we could drop in another implementation that would accumulate log messages in memory, compress them, and then upload them to S3. If the S3 connection were down or slow, the implementation would queue the files up on disk. Kind of like Facebook's Scribe.

My guess is that it would take me four or five days to write and test this. If there's another implementation out there, I'd love to know about it.

Answer 1

There is a plugin for fluentd that stores files to s3. (Fluentd is a nice "log file collector")

Read more about it here: https://docs.fluentd.org/output/s3

If the s3 connection is down or slow it will buffer the output for you.

Answer 2

You can try to write a custom appender for logback or log4j2 and use this appender in respective configuration.

This way, you don't have to write an entire logging framework, but only the part you need and use the rest from a working framework.

There are also a few of them on github. For instance: shuwada/logback-s3