Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is there a Logback Layout that Creates JSON Objects with Message Parameters as Attributes?

I want to send log events to Loggly as JSON objects with parameterized string messages. Our project currently has a lot of code that looks like this:

String someParameter = "1234"; logger.log("This is a log message with a parameter {}", someParameter); 

We're currently using Logback as our SLF4J backend, and Logback's JsonLayout to serialize our ILogEvent objects into JSON. Consequentially, by they time our log events are shipped to Loggly, they look like this:

{     "message": "This is a log message with a parameter 1234",     "level": INFO,     .... } 

While this does work, it sends a different message string for every value of someParameter, which renders Loggly's automatic filters next to useless.

Instead, I'd like to have a Layout that creates JSON that looks like this:

{     "message": "This is a log message with a parameter {}",     "level": INFO,     "parameters": [         "1234"     ] } 

This format would allow Loggly to group all log events with the message This is a log message with a parameter together, regardless of the value of someParameter.

It looks like Logstash's KV filter does something like this - is there any way to accomplish this task with Logback, short of writing my own layout that performs custom serialization of the ILogEvent object?

like image 467
MusikPolice Avatar asked Mar 24 '14 16:03

MusikPolice


People also ask

What is LogstashEncoder?

The LogstashEncoder writes all MDC entries to a separate JSON field. To show MDC entries in the classic console output, the appender layout pattern must be extended. The MDC uses a ThreadLocal, so it is important to clear it after exiting the thread.

What is encoder in Logback?

Encoders are responsible for transforming an event into a byte array as well as writing out that byte array into an OutputStream . Encoders were introduced in logback version 0.9. 19. In previous versions, most appenders relied on a layout to transform an event into a string and write it out using a java.

What is net logstash Logback?

Provides logback encoders, layouts, and appenders to log in JSON and other formats supported by Jackson. License. Apache 2.0MIT.


2 Answers

There is a JSON logstash encoder for Logback, logstash-logback-encoder

like image 140
Mark Roper Avatar answered Sep 19 '22 14:09

Mark Roper


So for me I was trying to log execution times, I created a pojo called ExecutionTime with name, method, class, duration.

I was then able to create it:

ExecutionTime time = new ExecutionTime("Controller Hit", methodName, className, sw.getTotalTimeMillis()); 

For logging I then used:

private final Logger logger = LoggerFactory.getLogger(this.getClass()); logger.info(append("metric", time), time.toString()); 

Make sure you have:

import static net.logstash.logback.marker.Markers.append; import org.slf4j.Logger; import org.slf4j.LoggerFactory; 

This will log something like this:

{      "ts":"2017-02-16T07:41:36.680-08:00",    "msg":"ExecutionTime [name=Controller Hit, method=setupSession, className=class com.xxx.services.controllers.SessionController, duration=3225]",    "logger":"com.xxx.services.metrics.ExecutionTimeLogger",    "level":"INFO",    "metric":{         "name":"Controller Hit",       "method":"setupSession",       "className":"class com.xxx.services.controllers.SessionController",       "duration":3225    } } 

Might be a different set up as I was using logback-spring.xml to output my logs to json:

<?xml version="1.0" encoding="UTF-8"?> <configuration>     <include resource="org/springframework/boot/logging/logback/base.xml"/>     <property name="PROJECT_ID" value="my_service"/>     <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">         <File>app/logs/${PROJECT_ID}.json.log</File>         <encoder class="net.logstash.logback.encoder.LogstashEncoder">             <fieldNames>                 <timestamp>ts</timestamp>                 <message>msg</message>                 <thread>[ignore]</thread>                 <levelValue>[ignore]</levelValue>                 <logger>logger</logger>                 <version>[ignore]</version>             </fieldNames>         </encoder>         <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">             <maxIndex>10</maxIndex>             <FileNamePattern>app/logs/${PROJECT_ID}.json.log.%i</FileNamePattern>         </rollingPolicy>         <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">             <MaxFileSize>20MB</MaxFileSize>         </triggeringPolicy>     </appender>     <logger name="com.xxx" additivity="false" level="DEBUG">         <appender-ref ref="FILE"/>         <appender-ref ref="CONSOLE"/>     </logger>     <root level="WARN">         <appender-ref ref="FILE"/>     </root> </configuration> 
like image 29
andrewps Avatar answered Sep 16 '22 14:09

andrewps