K8S, fluent-bit & elastic: Tried to parse field as object, but found a concrete value

Question

I switched from using fluentd to fluent-bit for ingesting K8S cluster logs into elastic, but now I am getting a lot of those error messages (showing up in kibana, so it works at least partially):

[object mapping for [kubernetes.labels.app] \
  tried to parse field [app] as object, \
  but found a concrete value, \
  object mapping for [kubernetes.labels.app] tried to parse field [app] as object, \
  but found a concrete value]

I don't really understand what that means, to be honest. I googled myself crazy, but found nothing of value. i imagine elastic expects "more dots below" the kubernetes.app keyword, but i fail to see why (is kubernetes.labels.app not valid together with kubernetes.labels.app.kubernetes.io??)

The closest thing I could come up with is a change of the kubernetes "property" in elastic to type text, yet I ...

• don't really know if this is the correct solution,
• failed to find out how,
• and would not know how to do this for all new logstash-* indexes which are automatically created by fluent-bit.

can someone help me out here? I am stuck.

I tried re-indexing into a new index, which at least did not fail, but didn't help (obviously, because the old indexes are still used, naturally). also, i have no clue whether the "new mapping" i created (see here) is correct. my fluent-bit config can be found in the same gist.

any help greatly appreciated.

Answer 1

When you push logs from k8s containers using an agent like fluentbit some documents have kubernetes.labels.app field as a "json object" type

While in some documents the kubernetes.labels.app field will be a "text" type field

kubernetes.labels.app : "app1"

An index can support mapping for only one type for each field. So if your mapping specifies that "kubernetes.labels.app" is an "object", the documents that have it as "text" type will not be processed and won't be stored. And vice versa.

Hence the error,

[object mapping for [kubernetes.labels.app] \
  tried to parse field [app] as object, \
  but found a concrete value, \
  object mapping for [kubernetes.labels.app] tried to parse field [app] as object, \
  but found a concrete value]

One way to tackle this is to use ingest pipelines in elasticsearch to rename the text field "kubernetes.labels.app" to a new nested sub-field like "kubernetes.labels.app.value". This is the if else mechanism you were asking.

I did this in few steps using dev console,

Step 1: (Create an ingest pipeline with a simple Rename processor).

Notice that in "if" I am checking if the field is of type String, and in the "target_field" I have set a new location where the value will be stored. Set "ignore_missing" and "ignore_failure" to "true" to avoid missing out any documents .

PUT _ingest/pipeline/rename_pipeline
{
  "processors": [
    {
      "rename": {
        "field": "kubernetes.labels.app",
        "target_field": "kubernetes.labels.app.value",
        "ignore_missing": true,
        "if": "ctx?.kubernetes?.labels?.app instanceof String",
        "tag": "ihd_dev",
        "ignore_failure": true
      }
    }
  ]
}

https://www.elastic.co/guide/en/elasticsearch/reference/8.14/rename-processor.html

Step 2: (Attach the ingest pipeline to an index or index template):

PUT _index_template/index1_template
{
  "index_patterns": ["index1-*"],
  "template": {
    "settings": {
      "index.default_pipeline": "rename_pipeline"
      ..
    }
  }
}