Send spring boot logs directly to logstash with no file

Question

So, I'm building a full cloud solution using kubernetes and spring boot.

My spring boot application is deployed to a container and logs directly on the console. As containers are ephemerals I'd like to send logs also to a remote logstash server, so that they can be processed and sent to elastic.

Normally I would install a filebeat on the server hosting my application, and I could, but isn't there any builtin method allowing me to avoid writing my log on a file before sending it?

Currently I'm using log4j but I see no problem in switching to another logger as long it has a "logbackappender".

Answer 1

You can try to add logback.xml in resources folder :

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE configuration>

<configuration scan="true">
    <include resource="org/springframework/boot/logging/logback/base.xml"/>

    <appender name="logstash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <param name="Encoding" value="UTF-8"/>
        <remoteHost>localhost</remoteHost>
        <port>5000</port>
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <customFields>{"app_name":"YourApp", "app_port": "YourPort"}</customFields>
        </encoder>
    </appender>

    <root level="INFO">
        <appender-ref ref="logstash"/>
    </root>

</configuration>

Then add logstash encoder dependency :

pom.xml

 <dependency>
        <groupId>net.logstash.logback</groupId>
        <artifactId>logstash-logback-encoder</artifactId>
        <version>4.11</version>
 </dependency>

logstash.conf

input {
    udp {
        port => "5000"
        type => syslog
        codec => json
    }
    tcp {
        port => "5000"
        type => syslog
        codec => json_lines
    }
    http {
        port => "5001"
        codec => "json"
    }
}

filter {
    if [type] == "syslog" {
        mutate {
            add_field => { "instance_name" => "%{app_name}-%{host}:%{app_port}" }
        }
    }
}

output {
    elasticsearch {
        hosts => ["${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}"]
        index => "logs-%{+YYYY.MM.dd}"
    }
}

Answer 2

I've just created a full working example in my repository Hope to be helpful for someone