What's the best practice to send logs to graylog from a JVM application which runs within a docker container?

Question

I'm using graylog as central logging server and i'm using a gelf log4j2-appender to send the log messages to graylog. This works fine. Now i created a docker image of my application and i'm able to run my software as docker container.

Using docker i also log to stdout (console-appender) to get the application logs into docker (docker logs {containerId}).

Now i ask myself wether i could spare on the gelf log4j2-appender and use instead a docker log-driver/plugin for gelf. (see https://docs.docker.com/engine/admin/logging/overview/)

What's the best practice here? I think using the docker log plugin would send the whole string message to graylog and graylog would need to extract the meta information from that string (so i need to provide this meta data within the log message, for example the log_level). This may cause more resource consumption on graylog side and it's also not possible to configure docker for sending only error messages to graylog. This leads to more network traffic. Using the log4j2 gelf-appender i'm able to provide some meta data additional to the log message without including it in the main log message and there wouldn't be an extraction needed on graylog side. It's also possible to configure which messages should be sent to graylog by log_level. Or am i wrong? What is the best solution or what are the pros and cons of each way of sending logs to graylog?

Answer 1

I'd recommend using an existing GELF appender for the logging framework you're using (e. g. logstash-gelf) instead of logging everything to stdout and use the GELF logging driver of Docker.

Using a proper GELF appender with a native Java logging framework enables you to use advanced features like an MDC to enrich you log messages with valuable structured information without having to re-parse those messages after receiving them on the server side. With the Docker GELF logging driver, you'd only receive log messages line-per-line, which especially with Java applications can be a headache to deal with (think multiline stack traces).

Most logging frameworks support static fields, so that you could "inject" the ID of your Docker container for example.