Permission issue when reading container logs with fluentd in Kubernetes

Question

I am really new to kubernetes and have testing app with redis and mongodb running in GCE. I would like to grap my log files with fluentd and send them to logz:

I use the following fluentd config file. I tested a similar version on my local machine.

<source>
    @type tail
    path /var/log/containers/squidex*.log
    pos_file /var/log/squidex.log.pos
    tag squidex.logs
    format json
</source>

<match squidex.logs>
    @type copy
    <store>
        @type logzio_buffered
        endpoint_url https://listener.logz.io:8071?token=...
        output_include_time true
        output_include_tags true
        buffer_type file
        buffer_path /fluentd/log/squidex.log.buffer
        flush_interval 10s
        buffer_chunk_limit 1m
    </store>
    <store>
        @type stdout
    </store>
</match>

My kubernetes configuration is:

---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd-logging
  labels:
    app: fluentd-logging
spec:
  template:
    metadata:
      labels:
        app: fluentd-logging
    spec:
      containers:
      - name: fluentd
        image: gcr.io/squidex-157415/squidex-fluentd:latest
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 40m
        volumeMounts:
        - name: varlog
          mountPath: /var/log
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log

Almost everything works, but when I run the fluentd pods I see the following entries in the log output from these pods:

2017-04-22T09:49:22.286740784Z 2017-04-22 09:49:22 +0000 [warn]: 
/var/log/containers/squidex-282724611-3nhtw_default_squidex-ed7c437e677d3438c137cdc80110d106339999a6ba8e495a5752fe6d5da9e70d.log unreadable. 
It is excluded and would be examined next time

How can I get permissions to those log files?

Answer 1

This is not a permission issue but broken symlinks. Kubernetes is using symbolic links from /var/log/containers to /var/log/pods to /var/lib/docker/containers. You can confirm this from any node of your cluster using ls -la

Your DaemonSet configuration should include something like:

volumeMounts:
- name: varlog
  mountPath: /var/log/
  readOnly: true
  - name: varlibdockercontainers
  mountPath: /var/lib/docker/containers
  readOnly: true
[...]
volumes:
- name: varlog
  hostPath:
    path: /var/log/
- name: varlibdockercontainers
  hostPath:
    path: /var/lib/docker/containers

This way, you are mounting both the logs files directory and the symlinks of symlinks so your fluentd can read everything.