How can fluent-bit add custom metadata to each event message being sent to splunk

Question

I'm using fluent-bit within Kubernetes to forward logs to Splunk. We'll be using the same Splunk index for multiple Kubernetes clusters, so I want to tag each event being forwarded from fluent-bit with the cluster that it comes from.

I tried using the modify functionality to "Add" or "Set" a new field in the event.

fluent-bit-filter.conf: |-
   [FILTER]
       Name                kubernetes
       Match               kube.*
       Kube_Tag_Prefix     kube.var.log.containers.
       Kube_URL            https://kubernetes.default.svc:443
       Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
       Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
       K8S-Logging.Parser  On
       K8S-Logging.Exclude On
       Add cluster devcluster

Sample log that I actually receive (missing the newly added field "cluster")

[305] kube.var.log.containers.calico-node-xzwnv_kube-system_calico-node-a4a6a2261a76ec419e9cf13ae39732b3e918726573cf1a0dece648e679011578.log: [1565578883.799679612, {"log"=>"2019-08-12 03:01:23.799 [INFO][68] int_dataplane.go 830: Received interface update msg=&intdataplane.ifaceUpdate{Name:"cali5d1a7318787", State:"up"}

Answer 1

Figured it out. You have to use a separate filter named modify. When using the helm chart, you need to add a section called rawConfig in the values.yaml and put your additional filter in between the @INCLUDE for fluent-bit-filter.conf

rawConfig: |-
 @INCLUDE fluent-bit-service.conf
 @INCLUDE fluent-bit-input.conf
 @INCLUDE fluent-bit-filter.conf
 [FILTER]
     Name modify
     Match *
     Add cluster devcluster
 @INCLUDE fluent-bit-output.conf