How to exclude namespace from fluent-bit logging

Question

Is there a way to exclude certain namespaces in fluent-bit? I would like to exclude certain namespaces, so that fluent-bit doesn't forward all logs created in those namespaces to ELK.

Is there a way to do it besides adding annotation to each pod in that namespace? I'm aware that you can update all of the pods annotations in a namespace via kubectl.

kubectl annotate pods --namespace=pks-system --all fluentbit.io/exclude='true'

Answer 1

I think the following input plugin configuration can do this:

 [INPUT]
        Name              tail
        Path              /var/log/containers/*.log
        Exclude_Path      /var/log/containers/*_<myappnamespace>_*.log
        Tag               kube.infra.<namespace_name>.<pod_name>.<container_name>
        Tag_Regex         (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
        Parser            cri
        DB                /var/log/flb_kube_infra.db
        Mem_Buf_Limit     500KB
        Skip_Long_Lines   On
        Refresh_Interval  10

Found it here: https://github.com/fluent/fluent-bit/issues/758

The Exclude_Path property defines the name of the namespace for which logs will be ignored.