How to get filebeat to ignore certain container logs

Question

I am using elasticserach 6.8 and filebeat 6.8.0 in a Kubernetes cluster. I want filebeat to ignore certain container logs but it seems almost impossible :).

This is my autodiscover config

filebeat.autodiscover:
  providers:
   - type: kubernetes
     hints.enabled: true
     templates:
      - condition:
        contains:
          kubernetes.namespace: bagmessage
        config:
         - type: docker
           containers.ids:
             - "${data.kubernetes.container.id}"
           processors:
             - drop_event:
                 when:
                   or: 
                     - contains:
                         kubernetes.container.name: "filebeat"
                     - contains:
                         kubernetes.container.name: "weave-npc"
                     - contains:
                         kubernetes.container.name: "bag-fluentd-es"
                     - contains:
                         kubernetes.container.name: "logstash"
                     - contains:
                         kubernetes.container.name: "billing"

I've tried many variations of this configuration but still filebeats is processing container logs that I want it to ignore.

I'd like to know if what I want to do is possible and if so, what am I doing wrong?

Thanks

Answer 1

The first error I see in your config is incorrect indentation of the condition section in the template. Should be:

   - type: kubernetes
     hints.enabled: true
     templates:
       - condition:
           contains:
             kubernetes.namespace: bagmessage

Secondly, I'm not sure the kubernetes.* is visible to the processors inside the config with type: docker. You may try to reference docker.container.name instead. Or alternatively you can move all your k8s-specific conditions to the condition section under the templates:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true
      templates:
        - condition:
            and:
              - contains.kubernetes.namespace: bagmessage
              - contains.container.name: billing
          config:
              ...

Also, make sure that "container.name" (but not "pod.name") is indeed what you want.