Kubernetes deployment - Externalizing log file

Question

I have a spring boot application with the below docker file.

FROM docker.com/base/jdk1.8:latest

MAINTAINER Application Engineering [ https://docker.com/ ]

RUN mkdir -p /opt/docker/svc

COPY application/weather-service.war /opt/docker/svc/

CMD java -jar /opt/docker/svc/weather-service.war --spring.config.location=file:/conf/application.properties -Dlogging.config=/conf/logback.xml

I can use kubernetes configMap or secrets for application.properties and use the volume mount option as below.

"spec": {
        "volumes": [
          {
            "name": "svc-prop",
            "configMap": {
              "name": "svc-app-config",
              "items": [
                {
                  "key": "application.properties",
                  "path": "application.properties"
                }
              ]
            }
          }
         ],
        "containers": [
          "volumeMounts": [
              {
                "name": "svc-prop",
                "mountPath": "/conf"
              }
         ]

How can i achieve the samething for logback.xml. Do i need to use secrets as a file in this case?

I dont want to bundle logback.xml file with the image as we might be changing the log level at runtime.

Is there any other better approach for keeping logback.xml for spring boot app in Kubernetes?

Answer 1

Usually you do not want to provide the whole logback.xml file but rather it's logger list which requires updating at runtime most frequently. In order to achieve this you can use Logback's file inclusion feature:

1. Write your logback.xml file as usual except logger list. Use include element instead:

    <configuration scan="true" scanPeriod="10 seconds" debug="true">
        <appender ...></appender>
        <root level="INFO">...</root>

        <!-- Import loggers configuration from external file -->
        <include file="config/mount/loggers-include.xml"/>
    </configuration>

Note those scan* attributes. They are essential for log config reloading at runtime.

2. Define all the loggers in Kubernetes ConfigMap with loggers-include.xml data section:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: microservice-loggers    # the name to refer to from deployment (see below)
     namespace: upc
   data:
     loggers-include.xml: |+
       <included>
         <logger name="org.springframework.cloud.netflix.zuul" level="INFO"/>
         <logger name="com.netflix.zuul" level="INFO"/>                              
         <logger name="com.netflix.hystrix" level="INFO"/>                          
         <logger name="com.netflix.ribbon" level="DEBUG"/>
         <logger name="com.netflix.loadbalancer" level="INFO"/>                     
       </included>
   

   Note that all the included content must be enclosed in included tag in order to be correctly parsed by Logback.

3. Mount your ConfigMap's data into container as config/mount/loggers-include.xml file:

   apiVersion: apps/v1
   kind: Deployment
   ...
   spec:
     ...
     template:
       ...
       spec:
         # declare the volume created from ConfigMap
         volumes:
           - name: config-volume             # used with this name below
             configMap:
               name: microservice-loggers    # declared in previous step
         containers:
         - name: microservice
           ...
           ports:
             ...
           # mount the volume declared above to container's file system
           volumeMounts:
             - mountPath: /microservice/config/mount
               name: config-volume       # declared above
   

   Note that mount directory must not be created neither by container itself nor its image. Moreover, if such a directory exists, all its content will be removed during mounting.

4. Apply the ConfigMap and run the declared deployment. To check if loggers file is correctly mounted execute the following command:

   $ kubectl exec restorun-7d757b7c6-wcslx -- ls -l /microservice/config/mount
   total 0
   lrwxrwxrwx    1 root     root            26 Aug 14 05:52 loggers-include.xml -> ..data/loggers-include.xml
   

   Also if you've set debug=true attribute in Logback's configuration element (see step 1) then you should see the following record in STDOUT during application startup:

   05:52:17,031 |-INFO in ch.qos.logback.core.joran.util.ConfigurationWatchListUtil@6e06451e - Adding [file:/microservice/config/mount/loggers-include.xml] to configuration watch list.
   

5. Now you can edit your ConfigMap (e.g. set com.netflix.hystrix to level WARN), save its file and tell Kubernetes to apply the changes to the application:

   $ kubectl apply -f microservice-log-configmap.yaml
   configmap "microservice-loggers" configured
   

   Again, Logback should reflect the changes by logging with the following message to standard output:

       05:59:16,974 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [com.netflix.hystrix] to WARN
   

   You can also check the effective logging level by directly asking it from the Spring Boot Actuator (if you have an access to this endpoint):

   $ curl http://<k8s-external-ip>/actuator/loggers/com.netflix.hystrix
   {
     "configuredLevel" : "WARN",
     "effectiveLevel" : "WARN"
   }
   

If the level stays the same, wait for a minute and check again: it takes some time for changes to propagate through Kubernetes and Logback. More info on this topic:

• Logback auto reloading
• Kubernetes ConfigMap mounting
  (see Mounted ConfigMaps are updated automatically section)