how to add custom field to log logback-logstash

Question

My spring boot log currently looks like the following.

{"@timestamp":"2018-08-07T14:49:21.244+01:00","@version":"1","message":"Starting Application on ipkiss bla bla)","logger_name":"logger name....","thread_name":"main","level":"INFO","level_value":20000}

with the logback-spring.xml setup like below

<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="com.ipkiss.correlate.logback.CorrelationPatternLayoutEncoder">
            <pattern>%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(%5p) %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} id = %id  %m%n%wEx</pattern>
        </encoder>
</appender>

and my class for LayoutEncoder looks like this

public class CorrelationPatternLayoutEncoder extends PatternLayoutEncoder {
    public CorrelationPatternLayoutEncoder() {

    }

@Override
public void start() {
    PatternLayout patternLayout = new PatternLayout();
    patternLayout.getDefaultConverterMap().put("id", CorrelationConverter.class.getName());
    patternLayout.setContext(context);
    patternLayout.setPattern(getPattern());
    patternLayout.setOutputPatternAsHeader(outputPatternAsHeader);
    patternLayout.start();
    this.layout = patternLayout;
    this.started = true;
}

}

what I was trying to achieve is to add the id to the log, I can't make logstach append my id, i tried Custom field according to the docs but I couldn't make it work, any ideas how I can achieve this?

this is what i want to end up with

{"id":"3a7ccd34-d66a-4fcc-a12e-763a395a496c","@timestamp":"2018-08-07T14:49:21.244+01:00","@version":"1","message":"Starting Application on ipkiss bla bla)","logger_name":"logger name....","thread_name":"main","level":"INFO","level_value":20000}

or id being appended at the end of the log.

Answer 1

From the logstash-encoder github page

By default, each entry in the Mapped Diagnostic Context (MDC) (org.slf4j.MDC) will appear as a field in the LoggingEvent.

So in short, if you add your id entry into MDC it will automatically be included in all of your logs.

To add your id to MDC do the following:

MDC.put("id", uuid);

As MDC is a thread local variable you will have to clear it after your request has finished using

MDC.remove("id")

In a web application, add and clearing the values in MDC is usually done in a servlet filter ie.

public class IdFilter implements Filter{
    @Override
    public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain) throws IOException, ServletException {
        MDC.put("id", UUID.randomUUID().toString());
        try {
            filterChain.doFilter(servletRequest, servletResponse);
        } finally {
            MDC.remove("id");
        }
    }
}

Answer 2

You can add custom log field by Creating a custom conversion specifier. First off all create a custom converter class which extends ClassicConverter class.

import ch.qos.logback.classic.pattern.ClassicConverter;
import ch.qos.logback.classic.spi.ILoggingEvent;
import java.util.UUID;

public class LogUniqueIdConverter extends ClassicConverter {

    @Override
    public String convert(ILoggingEvent event) {
        return String.valueOf(UUID.randomUUID());
    }
}

ClassicConverter objects are responsible for extracting information out of ILoggingEvent instances and producing a String.

So just write in your logback configuration file so that logback know about the new Converter. For this you just need to declare the new conversion word in the configuration file, as shown below:

<configuration>

  <conversionRule conversionWord="customId" 
                  converterClass="com.naib.util.MySampleConverter" />
        
  <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
      <pattern>{"id": "%customId" %msg%n%throwable %msg%n}</pattern>
    </encoder>
  </appender>

  <root level="DEBUG">
    <appender-ref ref="STDOUT" />
  </root>
</configuration>