Fluentd pods running via daemonset getting terminated with warnings on google container engine

Question

I am trying to run fluentd as a daemonset on kubernetes cluster (GKE). The config is getting parsed successfully, then the plugins are receiving shutdown signal with few warn messages. There are no error messages. I tried increasing the verbosity level and following is the output of pods:

fluentd-7przp fluentd 2018-09-08 11:02:46 +0000 [info]: #0 fluent/log.rb:322:info: starting fluentd worker pid=9 ppid=1 worker=0
fluentd-7przp fluentd 2018-09-08 11:02:46 +0000 [info]: #0 fluent/log.rb:322:info: fluentd worker is now running worker=0
fluentd-sr764 fluentd 2018-09-08 11:02:50 +0000 [warn]: #0 fluent/log.rb:342:warn: dump an error event: error_class=NoMethodError error="undefined method `[]' for nil:NilClass" location="/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'" tag="fluent.info" time=2018-09-08 11:02:45.151774166 +0000 record={"worker"=>0, "message"=>"fluentd worker is now running worker=0"}
fluentd-dhwnx fluentd 2018-09-08 11:02:51 +0000 [warn]: #0 fluent/log.rb:342:warn: dump an error event: error_class=NoMethodError error="undefined method `[]' for nil:NilClass" location="/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'" tag="fluent.info" time=2018-09-08 11:02:46.029522363 +0000 record={"worker"=>0, "message"=>"fluentd worker is now running worker=0"}
fluentd-7przp fluentd 2018-09-08 11:02:51 +0000 [warn]: #0 fluent/log.rb:342:warn: dump an error event: error_class=NoMethodError error="undefined method `[]' for nil:NilClass" location="/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'" tag="fluent.info" time=2018-09-08 11:02:46.538377182 +0000 record={"worker"=>0, "message"=>"fluentd worker is now running worker=0"}
fluentd-sr764 fluentd 2018-09-08 11:02:55 +0000 [warn]: #0 fluent/log.rb:342:warn: dump an error event: error_class=NoMethodError error="undefined method `[]' for nil:NilClass" location="/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'" tag="fluent.warn" time=2018-09-08 11:02:50.153922217 +0000 record={"error"=>"#<NoMethodError: undefined method `[]' for nil:NilClass>", "location"=>"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'", "tag"=>"fluent.info", "time"=>2018-09-08 11:02:45.151774166 +0000, "record"=>{"worker"=>0, "message"=>"fluentd worker is now running worker=0"}, "message"=>"dump an error event: error_class=NoMethodError error=\"undefined method `[]' for nil:NilClass\" location=\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\" tag=\"fluent.info\" time=2018-09-08 11:02:45.151774166 +0000 record={\"worker\"=>0, \"message\"=>\"fluentd worker is now running worker=0\"}"}
fluentd-sr764 fluentd 2018-09-08 11:03:10 +0000 [warn]: #0 fluent/log.rb:342:warn: dump an error event: error_class=NoMethodError error="undefined method `[]' for nil:NilClass" location="/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'" tag="fluent.warn" time=2018-09-08 11:03:05.168427649 +0000 record={"error"=>"#<NoMethodError: undefined method `[]' for nil:NilClass>", "location"=>"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'", "tag"=>"fluent.warn", "time"=>2018-09-08 11:03:00.165843014 +0000, "record"=>{"error"=>"#<NoMethodError: undefined method `[]' for nil:NilClass>", "location"=>"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'", "tag"=>"fluent.warn", "time"=>2018-09-08 11:02:55.156840516 +0000, "record"=>{"error"=>"#<NoMethodError: undefined method `[]' for nil:NilClass>", "location"=>"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'", "tag"=>"fluent.warn", "time"=>2018-09-08 11:02:50.153922217 +0000, "record"=>{"error"=>"#<NoMethodError: undefined method `[]' for nil:NilClass>", "location"=>"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'", "tag"=>"fluent.info", "time"=>2018-09-08 11:02:45.151774166 +0000, "record"=>{"worker"=>0, "message"=>"fluentd worker is now running worker=0"}, "message"=>"dump an error event: error_class=NoMethodError error=\"undefined method `[]' for nil:NilClass\" location=\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\" tag=\"fluent.info\" time=2018-09-08 11:02:45.151774166 +0000 record={\"worker\"=>0, \"message\"=>\"fluentd worker is now running worker=0\"}"}, "message"=>"dump an error event: error_class=NoMethodError error=\"undefined method `[]' for nil:NilClass\" location=\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\" tag=\"fluent.warn\" time=2018-09-08 11:02:50.153922217 +0000 record={\"error\"=>\"#<NoMethodError: undefined method `[]' for nil:NilClass>\", \"location\"=>\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\", \"tag\"=>\"fluent.info\", \"time\"=>2018-09-08 11:02:45.151774166 +0000, \"record\"=>{\"worker\"=>0, \"message\"=>\"fluentd worker is now running worker=0\"}, \"message\"=>\"dump an error event: error_class=NoMethodError error=\\\"undefined method `[]' for nil:NilClass\\\" location=\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\" tag=\\\"fluent.info\\\" time=2018-09-08 11:02:45.151774166 +0000 record={\\\"worker\\\"=>0, \\\"message\\\"=>\\\"fluentd worker is now running worker=0\\\"}\"}"}, "message"=>"dump an error event: error_class=NoMethodError error=\"undefined method `[]' for nil:NilClass\" location=\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\" tag=\"fluent.warn\" time=2018-09-08 11:02:55.156840516 +0000 record={\"error\"=>\"#<NoMethodError: undefined method `[]' for nil:NilClass>\", \"location\"=>\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\", \"tag\"=>\"fluent.warn\", \"time\"=>2018-09-08 11:02:50.153922217 +0000, \"record\"=>{\"error\"=>\"#<NoMethodError: undefined method `[]' for nil:NilClass>\", \"location\"=>\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\", \"tag\"=>\"fluent.info\", \"time\"=>2018-09-08 11:02:45.151774166 +0000, \"record\"=>{\"worker\"=>0, \"message\"=>\"fluentd worker is now running worker=0\"}, \"message\"=>\"dump an error event: error_class=NoMethodError error=\\\"undefined method `[]' for nil:NilClass\\\" location=\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\" tag=\\\"fluent.info\\\" time=2018-09-08 11:02:45.151774166 +0000 record={\\\"worker\\\"=>0, \\\"message\\\"=>\\\"fluentd worker is now running worker=0\\\"}\"}, \"message\"=>\"dump an error event: error_class=NoMethodError error=\\\"undefined method `[]' for nil:NilClass\\\" location=\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\" tag=\\\"fluent.warn\\\" time=2018-09-08 11:02:50.153922217 +0000 record={\\\"error\\\"=>\\\"#<NoMethodError: undefined method `[]' for nil:NilClass>\\\", \\\"location\\\"=>\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\", \\\"tag\\\"=>\\\"fluent.info\\\", \\\"time\\\"=>2018-09-08 11:02:45.151774166 +0000, \\\"record\\\"=>{\\\"worker\\\"=>0, \\\"message\\\"=>\\\"fluentd worker is now running worker=0\\\"}, \\\"message\\\"=>\\\"dump an error event: error_class=NoMethodError error=\\\\\\\"undefined method `[]' for nil:NilClass\\\\\\\" location=\\\\\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\\\\\" tag=\\\\\\\"fluent.info\\\\\\\" time=2018-09-08 11:02:45.151774166 +0000 record={\\\\\\\"worker\\\\\\\"=>0, \\\\\\\"message\\\\\\\"=>\\\\\\\"fluentd worker is now running worker=0\\\\\\\"}\\\"}\"}"}, "message"=>"dump an error event: error_class=NoMethodError error=\"undefined method `[]' for nil:NilClass\" location=\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\" tag=\"fluent.warn\" time=2018-09-08 11:03:00.165843014 +0000 record={\"error\"=>\"#<NoMethodError: undefined method `[]' for nil:NilClass>\", \"location\"=>\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\", \"tag\"=>\"fluent.warn\", \"time\"=>2018-09-08 11:02:55.156840516 +0000, \"record\"=>{\"error\"=>\"#<NoMethodError: undefined method `[]' for nil:NilClass>\", \"location\"=>\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\", \"tag\"=>\"fluent.warn\", \"time\"=>2018-09-08 11:02:50.153922217 +0000, \"record\"=>{\"error\"=>\"#<NoMethodError: undefined method `[]' for nil:NilClass>\", \"location\"=>\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\", \"tag\"=>\"fluent.info\", \"time\"=>2018-09-08 11:02:45.151774166 +0000, \"record\"=>{\"worker\"=>0, \"message\"=>\"fluentd worker is now running worker=0\"}, \"message\"=>\"dump an error event: error_class=NoMethodError error=\\\"undefined method `[]' for nil:NilClass\\\" location=\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\" tag=\\\"fluent.info\\\" time=2018-09-08 11:02:45.151774166 +0000 record={\\\"worker\\\"=>0, \\\"message\\\"=>\\\"fluentd worker is now running worker=0\\\"}\"}, \"message\"=>\"dump an error event: error_class=NoMethodError error=\\\"undefined method `[]' for nil:NilClass\\\" location=\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\" tag=\\\"fluent.warn\\\" time=2018-09-08 11:02:50.153922217 +0000 record={\\\"error\\\"=>\\\"#<NoMethodError: undefined method `[]' for nil:NilClass>\\\", \\\"location\\\"=>\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\", \\\"tag\\\"=>\\\"fluent.info\\\", \\\"time\\\"=>2018-09-08 11:02:45.151774166 +0000, \\\"record\\\"=>{\\\"worker\\\"=>0, \\\"message\\\"=>\\\"fluentd worker is now running worker=0\\\"}, \\\"message\\\"=>\\\"dump an error event: error_class=NoMethodError error=\\\\\\\"undefined method `[]' for nil:NilClass\\\\\\\" location=\\\\\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\\\\\" tag=\\\\\\\"fluent.info\\\\\\\" time=2018-09-08 11:02:45.151774166 +0000 record={\\\\\\\"worker\\\\\\\"=>0, \\\\\\\"message\\\\\\\"=>\\\\\\\"fluentd worker is now running worker=0\\\\\\\"}\\\"}\"}, \"message\"=>\"dump an error event: error_class=NoMethodError error=\\\"undefined method `[]' for nil:NilClass\\\" location=\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\" tag=\\\"fluent.warn\\\" time=2018-09-08 11:02:55.156840516 +0000 record={\\\"error\\\"=>\\\"#<NoMethodError: undefined method `[]' for nil:NilClass>\\\", \\\"location\\\"=>\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\", \\\"tag\\\"=>\\\"fluent.warn\\\", \\\"time\\\"=>2018-09-08 11:02:50.153922217 +0000, \\\"record\\\"=>{\\\"error\\\"=>\\\"#<NoMethodError: undefined method `[]' for nil:NilClass>\\\", \\\"location\\\"=>\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\", \\\"tag\\\"=>\\\"fluent.info\\\", \\\"time\\\"=>2018-09-08 11:02:45.151774166 +0000, \\\"record\\\"=>{\\\"worker\\\"=>0, \\\"message\\\"=>\\\"fluentd worker is now running worker=0\\\"}, \\\"message\\\"=>\\\"dump an error event: error_class=NoMethodError error=\\\\\\\"undefined method `[]' for nil:NilClass\\\\\\\" location=\\\\\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\\\\\" tag=\\\\\\\"fluent.info\\\\\\\" time=2018-09-08 11:02:45.151774166 +0000 record={\\\\\\\"worker\\\\\\\"=>0, \\\\\\\"message\\\\\\\"=>\\\\\\\"fluentd worker is now running worker=0\\\\\\\"}\\\"}, \\\"message\\\"=>\\\"dump an error event: error_class=NoMethodError error=\\\\\\\"undefined method `[]' for nil:NilClass\\\\\\\" location=\\\\\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\\\\\" tag=\\\\\\\"fluent.warn\\\\\\\" time=2018-09-08 11:02:50.153922217 +0000 record={\\\\\\\"error\\\\\\\"=>\\\\\\\"#<NoMethodError: undefined method `[]' for nil:NilClass>\\\\\\\", \\\\\\\"location\\\\\\\"=>\\\\\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\\\\\", \\\\\\\"tag\\\\\\\"=>\\\\\\\"fluent.info\\\\\\\", \\\\\\\"time\\\\\\\"=>2018-09-08 11:02:45.151774166 +0000, \\\\\\\"record\\\\\\\"=>{\\\\\\\"worker\\\\\\\"=>0, \\\\\\\"message\\\\\\\"=>\\\\\\\"fluentd worker is now running worker=0\\\\\\\"}, \\\\\\\"message\\\\\\\"=>\\\\\\\"dump an error event: error_class=NoMethodError error=\\\\\\\\\\\\\\\"undefined method `[]' for nil:NilClass\\\\\\\\\\\\\\\" location=\\\\\\\\\\\\\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\\\\\\\\\\\\\" tag=\\\\\\\\\\\\\\\"fluent.info\\\\\\\\\\\\\\\" time=2018-09-08 11:02:45.151774166 +0000 record={\\\\\\\\\\\\\\\"worker\\\\\\\\\\\\\\\"=>0, \\\\\\\\\\\\\\\"message\\\\\\\\\\\\\\\"=>\\\\\\\\\\\\\\\"fluentd worker is now running worker=0\\\\\\\\\\\\\\\"}\\\\\\\"}\\\"}\"}"}
fluentd-7przp fluentd 2018-09-08 11:03:24 +0000 [debug]: #0 fluent/log.rb:302:debug: preparing shutdown output plugin type=:elasticsearch_dynamic plugin_id="kubelet_out_es"
fluentd-7przp fluentd 2018-09-08 11:03:24 +0000 [info]: #0 fluent/log.rb:322:info: shutting down output plugin type=:elasticsearch_dynamic plugin_id="kubelet_out_es"
fluentd-dhwnx fluentd 2018-09-08 11:03:25 +0000 [warn]: #0 fluent/log.rb:342:warn: dump an error event: error_class=NoMethodError error="undefined method `[]' for nil:NilClass" location="/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'" tag="fluent.debug" time=2018-09-08 11:03:24.151685730 +0000 record={"message"=>"fluentd main process get SIGTERM"}
fluentd-sr764 fluentd-dhwnx fluentd-7przp fluentd fluentd 2018-09-08 11:03:25 +0000 [debug]: #0 fluent/log.rb:302:debug: calling terminate on filter plugin type=:parser plugin_id="myapp_filter"
fluentd 2018-09-08 11:03:25 +0000 [debug]: #0 fluent/log.rb:302:debug: calling terminate on filter plugin type=:parser plugin_id="myapp_filter"
fluentd-dhwnxfluentd-sr764  fluentd 2018-09-08 11:03:25 +0000 [info]: fluent/log.rb:322:info: Worker 0 finished with status 0
fluentd 2018-09-08 11:03:25 +0000 [info]: fluent/log.rb:322:info: Worker 0 finished with status 0
2018-09-08 11:03:25 +0000 [debug]: #0 fluent/log.rb:302:debug: calling terminate on output plugin type=:elasticsearch_dynamic plugin_id="kubelet_out_es"
fluentd-7przp fluentd 2018-09-08 11:03:25 +0000 [debug]: #0 fluent/log.rb:302:debug: calling terminate on output plugin type=:elasticsearch_dynamic plugin_id="apiserver_out_es"
fluentd-7przp fluentd 2018-09-08 11:03:25 +0000 [debug]: #0 fluent/log.rb:302:debug: calling terminate on output plugin type=:elasticsearch_dynamic plugin_id="out_es"
fluentd-7przp fluentd 2018-09-08 11:03:25 +0000 [debug]: #0 fluent/log.rb:302:debug: calling terminate on filter plugin type=:parser plugin_id="myapp_filter"
fluentd-7przp fluentd 2018-09-08 11:03:26 +0000 [info]: fluent/log.rb:322:info: Worker 0 finished with status 0

fluentd.conf:

@include systemd.conf
@include kubernetes.conf

# Start of fluent.conf
<filter kubernetes.var.log.containers.myapp-provider**.log>
  @type parser
  @id myapp_filter
  key_name log
  reserve_data true
  remove_key_name_field true
  <parse>
    @type multiline
    format_firstline /^[A-Z]/
    format1 /^(?<level>[^ ]*)[ \t]+\[(?<time>[^\]]*)\] \[(?<thread>[^\]]*)\] \[(?<request>[^\]]*)\] (?<class>[^ ]*): (?<message>.*)$/
    time_format %Y-%m-%d %H:%M:%S,%L %z
  </parse>
</filter>

<match kubelet>
   @type elasticsearch_dynamic
   @id kubelet_out_es
   log_level info
   include_tag_key true
   host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
   port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
   scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
   ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"
   #user "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
   #password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"
   reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'true'}"
   index_name fluentd-${tag_parts[0]+ "-" + Time.at(time).getlocal("+05:30").strftime(@logstash_dateformat)}
   include_timestamp true
   #logstash_prefix "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash'}"
   #logstash_format true
   <buffer>
     flush_thread_count 8
     flush_interval 5s
     chunk_limit_size 2M
     queue_limit_length 32
     retry_max_interval 30
     retry_forever true
   </buffer>
</match>

<match kube-apiserver>
   @type elasticsearch_dynamic
   @id apiserver_out_es
   log_level info
   include_tag_key true
   host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
   port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
   scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
   ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"
   #user "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
   #password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"
   reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'true'}"
   index_name fluentd-${tag_parts[0]+ "-" + Time.at(time).getlocal("+05:30").strftime(@logstash_dateformat)}
   include_timestamp true
   #logstash_prefix "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash'}"
   #logstash_format true
   <buffer>
     flush_thread_count 8
     flush_interval 5s
     chunk_limit_size 2M
     queue_limit_length 32
     retry_max_interval 30
     retry_forever true
   </buffer>
</match>

<match **>
   @type elasticsearch_dynamic
   @id out_es
   log_level info
   include_tag_key true
   host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
   port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
   scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
   ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"
   #user "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
   #password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"
   reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'true'}"
   index_name fluentd-${record['kubernetes']['container_name']}-${Time.at(time).getlocal("+05:30").strftime(@logstash_dateformat)}
   include_timestamp true
   #logstash_prefix "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash'}"
   #logstash_format true
   <buffer>
     flush_thread_count 8
     flush_interval 5s
     chunk_limit_size 2M
     queue_limit_length 32
     retry_max_interval 30
     retry_forever true
   </buffer>
</match>

Can anyone please help to debug this. Thanks in advance!

Answer 1

This could be a kubernetes bug as mentioned here - https://github.com/kubernetes/kubernetes/issues/51376

Although this issue is closed as of now, I would suggest you to try running the same daemonset in default namespace. Even I tried running it in kube-system namespace but its failing and was running successfully in the default namespace.