How to capture multiple log streams from a single Docker container?

Question

Let's say I have a Java web app deployed in Docker, with several components producing logs within the container:

• Web application itself, logging to STDOUT and STDERR
• Application server, logging to server.log
• JVM, logging GC events to gc.log
• Others, e.g. underlying Linux OS

The application logs are routed to Splunk, but how should we retrieve logs for other components?

It seems there are at least two options:

• Log everything to STDOUT. Not ideal as this mixes logs and produces output that is more complex to separate, as different components produce logs in different formats.
• Use sidecar containers for each process other than the main process, and have those sidecars retrieve the logs from e.g. server.log and gc.log, and pipe them to STDOUT. Better, but this can have a significant impact on disk usage.

Is there a better solution?

Answer 1

TLDR; Don't mix log streams, using sidecar is the better solution and doesn't have "significant impact" on disk usage. Use sidecars to redirect logs to stdout and lean on the container ecosystem's capability to read individual container's log.

Note: Sidecar with log redirection logic does not have significant overhead. I'm quoting Kubernetes documentation in support of this - "The logic behind redirecting logs is minimal, so it's hardly a significant overhead"

---

The key is not to mix-up the log streams, as separating single stream from mixed log stream later could be tedious, if not impossible (with log events like exception stack trace which span multiple lines). For instance, if you mix up log streams, to analyze GC you would need to separate out GC log events alone, before feeding it into your tool of choice.

Solution is to use side-car pattern, detailed with example here

Follow Single Responsibility Principle. Have each sidecar tail one log file and stream it to stdout. To achieve this, all the sidecars should have access to webapp container's log volume mount (say /var/log) and the webapp should write all its log to shared volume mount(s).

The webapp will write its own log to stdout (Say through ConsoleAppender of LogBack or Log4j) and any other additional log files under shared volume mount (say /var/log). As an example, for GC log, set the JVM argument, to have the log generate beneath shared volume mount, -Xloggc:/var/log/mywebapp/gc.log.

For each additional log file, you would need a sidecar tailing (redirecting) log to stdout. So in your case you would need three sidecars, one for gc.log, one for server.log and another for /var/log/syslog

A sample sidecar container definition excerpt is given below.

  - name: count-log-1
    image: busybox
    args: [/bin/sh, -c, 'tail -n+1 -f /var/log/mywebapp/gc.log']
    volumeMounts:
    - name: varlog
      mountPath: /var/log

The above example shows a single sidecar definition redirecting/tailing log file from /var/log shared volume mount to stdout. This example was copied from this Kubernetes documentation.