Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How does multi-line logging work in Lambda -> CloudWatch

My multi-line logging events all end up multi-events - one event per line. According to the documentation:

Each call to LambdaLogger.log() results in a CloudWatch Logs event...

but then:

However, note that AWS Lambda treats each line returned by System.out and System.err as a separate event.

Looking inside LambdaAppender's source code, it seems that it proceeds to log the event to System.out anyway. So does that mean multi-line messages will always be broken down into multiple event?

I have read about configuring the multi_line_start_pattern, but that seems only applicable when you get to deploy a log agent, which isn't accessible in Lambda.

[Edit] LambdaAppender logs to LambdaLogger which logs to System.out.

[Edit] I found some post where a workaround was suggested - use '\r' for the eol when printing the messages. This seems to work for messages that my code produces. Stack traces logged everywhere are still a problem.

[Edit] I have been using two workarounds:

  1. Log complex data structures (e.g. sizable maps) in JSON. CloudWatch actually recognizes JSON strings in log events, and pretty print them.
  2. Replace '\n' with '\r'. For stack traces I created a utility method (this is in Kotlin, but the idea is generic enough):

    fun formatThrowable(t: Throwable): String {
        val buffer = StringWriter()
        t.printStackTrace(PrintWriter(buffer))
        return buffer.toString().replace("\n", "\r")
    }
    

I think in the long run a more ideal solution would be an Appender implementation that decorates ConsoleAppender, which would do the \r replacement on all messages passing through.

like image 784
jingx Avatar asked Nov 23 '17 00:11

jingx


2 Answers

Best practice is to use json in your logs. Instead of sending multiline outputs, send a formatted json (regardless of the language you are using, you will find a lib that already does that for you)

You will be amazed how easy it gets to browse your logs from there. For instance, aws cloudwatch insights automatically detects your fields, it allow to parse them and query them within seconds

like image 140
aherve Avatar answered Sep 16 '22 13:09

aherve


I suggest to use the project slf4j-simple-lambda and to refer to this blog for more explanations.

Using slf4j and slf4j-simple-lambda is solving elegantly your problem and the solution stay lightweight. The project includes the usage of the parameter org.slf4j.simpleLogger.newlineMethod which is there to solve this problem. By default, its value is auto and should be able to detect automatically the need for manual newline handling.

Discloser: I am co-author of slf4j-simple-lambda and author of the blog.

like image 37
Frank AFRIAT Avatar answered Sep 18 '22 13:09

Frank AFRIAT