Fluentd logs is full of backslash and kibana doesn't show k8s pods logs

Question

I set up an EFK stack for gathering my different k8s pods logs based on this tutorial on a Microk8s single node cluster. Everything is up and working and I can connect kibanna to elasticsearch and see the indexes but in the discovery section of kibana there is no log related to my pods and there are kubelete logs.

When I checked the logs of fluentd I saw that it is full of backslashes:

2019-08-05 15:23:17 +0000 [warn]: #0 [in_tail_container_logs] pattern not match: "2019-08-05T17:23:10.167379794+02:00 stdout P 2019-08-05 15:23:10 +0000 [warn]: #0 [in_tail_container_logs] pattern not match: \"2019-08-05T17:23:07.09726655+02:00 stdout P 2019-08-05 15:23:07 +0000 [warn]: #0 [in_tail_container_logs] pattern not match: \\\"2019-08-05T17:23:04.433817307+02:00 stdout P 2019-08-05 15:23:04 +0000 [warn]: #0 [in_tail_container_logs] pattern not match: \\\\\\\"2019-08-05T17:22:52.546188522+02:00 stdout P 2019-08-05 15:22:52 +0000 [warn]: #0 [in_tail_container_logs] pattern not match: \\\\\\\\\\\\\\\"2019-08-05T17:22:46.694679863+02:00 stdout F 
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\

Can someone please tell me what I am doing wrong and where all those backslashes are coming from?

Answer 1

For posterity, I'm going to provide this solution, which seems a lot more straight forward than anything else I've found:

env:
    - name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
      value: /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/

Just place that in the env variables underneath your fluentd YAML configuration.