Fluentd: pattern not matched error after migrating from kubernetes from 1.23 to .125

Question

The below container.conf works fine in Kubernetes 1.23 but fails after migrating to 1.25. I have also specified the deamonset that I have used to push the logs to cloudwatch. When I look into the logs of the fluentd deamonset I could see a lot of below errors

2023-04-03 01:32:06 +0000 [warn]: #0 [in_tail_container_logs] pattern not matched: "2023-04-03T01:32:02.9256618Z stdout F [2023-04-03T01:32:02.925Z] DEBUG transaction-677fffdfc4-tc4rx-18/TRANSPORTER: NATS client pingTimer: 1"

container.conf
==============
    <source>
      @type tail
      @id in_tail_container_logs
      @label @containers
      path /var/log/containers/*.log
      exclude_path ["/var/log/containers/fluentd*"]
      pos_file /var/log/fluentd-containers.log.pos
      tag *
      read_from_head true
      <parse>
        @type json
        time_format %Y-%m-%dT%H:%M:%S.%NZ
      </parse>
    </source>
    <label @containers>
      <filter **>
        @type kubernetes_metadata
        @id filter_kube_metadata
      </filter>
      <filter **>
        @type record_transformer
        @id filter_containers_stream_transformer
        <record>
          stream_name ${tag_parts[3]}
        </record>
      </filter>
      <match **>
        @type cloudwatch_logs
        @id out_cloudwatch_logs_containers
        region "#{ENV.fetch('AWS_REGION')}"
        log_group_name "/k8s-nest/#{ENV.fetch('AWS_EKS_CLUSTER_NAME')}/containers"
        log_stream_name_key stream_name
        remove_log_stream_name_key true
        auto_create_stream true
        <buffer>
          flush_interval 5
          chunk_limit_size 2m
          queued_chunks_limit_size 32
          retry_forever true
        </buffer>
      </match>
    </label>

Deamonset
==========
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    k8s-app: fluentd-cloudwatch
  name: fluentd-cloudwatch
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-cloudwatch
  template:
    metadata:
      labels:
        k8s-app: fluentd-cloudwatch
      annotations:
        iam.amazonaws.com/role: fluentd
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      containers:
        - env:
            - name: AWS_REGION
              value: us-west-1
            - name: AWS_EKS_CLUSTER_NAME
              value: dex-eks-west
          #image: 'fluent/fluentd-kubernetes-daemonset:v1.1-debian-cloudwatch'
          image: 'fluent/fluentd-kubernetes-daemonset:v1.15.3-debian-cloudwatch-1.1'
          imagePullPolicy: IfNotPresent
          name: fluentd-cloudwatch
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 200Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /config-volume
              name: config-volume
            - mountPath: /fluentd/etc
              name: fluentdconf
            - mountPath: /var/log
              name: varlog
            - mountPath: /var/lib/docker/containers
              name: varlibdockercontainers
              readOnly: true
            - mountPath: /run/log/journal
              name: runlogjournal
              readOnly: true
      dnsPolicy: ClusterFirst
      initContainers:
        - command:
            - sh
            - '-c'
            - cp /config-volume/..data/* /fluentd/etc
          image: busybox
          imagePullPolicy: Always
          name: copy-fluentd-config
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /config-volume
              name: config-volume
            - mountPath: /fluentd/etc
              name: fluentdconf
      terminationGracePeriodSeconds: 30
      volumes:
        - configMap:
            defaultMode: 420
            name: fluentd-config
          name: config-volume
        - emptyDir: {}
          name: fluentdconf
        - hostPath:
            path: /var/log
            type: ''
          name: varlog
        - hostPath:
            path: /var/lib/docker/containers
            type: ''
          name: varlibdockercontainers
        - hostPath:
            path: /run/log/journal
            type: ''
          name: runlogjournal

Answer 1

I had the same problem a while ago.

It seems to be an issue between the logs being emitted from the container and what is being written to the log file. Something is prefixing all logs with the <stdout/stderr> <?>

Ref. https://github.com/fluent/fluentd-kubernetes-daemonset/issues/434#issuecomment-747173567

Try following the discussion in the link I pasted you above; I solved it like this:

  <parse>
    @type regexp
    expression /^(?<time>.+) (?<stream>stdout|stderr)( (?<logtag>.))? (?<log>.*)$/
  </parse>