Increase the maximum size of log messages

Question

I would like to increase the maximum size of message saved in graylog with elasticsearch.

The maximum message size is 32 kb

So I update the mapping to remove the index on full_message:

graylog-custom-mapping.json:

{
  "template": "graylog_*",
  "mappings": {
    "message": {
      "properties": {
        "full_message": {
          "index": "no",
          "doc_values": false,
          "type": "string"
        }
      }
    }
  }
}

curl -X PUT -d @'graylog-custom-mapping.json' 'http://localhost:9200/_template/graylog-custom-mapping?pretty'

{
  "acknowledged" : true
}

I created a new Graylog index (graylog_5) from Graylog interface (System > Indices > Maintenance > Manually cycle deflector)

But my mapping seems to not be concidered:

curl -X GET 'http://localhost:9200/graylog_5/_mapping/message'

...
"full_message": {
    "type": "string",
    "analyzer": "standard"
},
...

My active template:

{
  "graylog-internal": {
    "order": -2147483648,
    "template": "graylog_*",
    "settings": {
      "index": {
        "analysis": {
          "analyzer": {
            "analyzer_keyword": {
              "filter": "lowercase",
              "tokenizer": "keyword"
            }
          }
        }
      }
    },
    "mappings": {
      "message": {
        "_source": {
          "enabled": true
        },
        "dynamic_templates": [
          {
            "internal_fields": {
              "mapping": {
                "index": "not_analyzed",
                "type": "string"
              },
              "match": "gl2_*"
            }
          },
          {
            "store_generic": {
              "mapping": {
                "index": "not_analyzed"
              },
              "match": "*"
            }
          }
        ],
        "properties": {
          "full_message": {
            "analyzer": "standard",
            "index": "analyzed",
            "type": "string"
          },
          "streams": {
            "index": "not_analyzed",
            "type": "string"
          },
          "source": {
            "analyzer": "analyzer_keyword",
            "index": "analyzed",
            "type": "string"
          },
          "message": {
            "analyzer": "standard",
            "index": "analyzed",
            "type": "string"
          },
          "timestamp": {
            "format": "yyyy-MM-dd HH:mm:ss.SSS",
            "type": "date"
          }
        }
      }
    },
    "aliases": {}
  },
  "graylog-custom-mapping": {
    "order": 0,
    "template": "graylog_*",
    "settings": {},
    "mappings": {
      "message": {
        "properties": {
          "full_message": {
            "index": "no",
            "type": "string",
            "doc_values": false
          }
        }
      }
    },
    "aliases": {}
  }
}

What's wrong with my configuration ?

Graylog 2.1.2 + ES 2.4.2

I have the following logs:

[2018-02-16 16:26:36,598][INFO ][cluster.metadata         ] [Zero] [graylog_5] creating index, cause [api], templates [graylog-internal, graylog-custom-mapping], shards [4]/[0], mappings [message]
[2018-02-16 16:26:37,091][INFO ][cluster.routing.allocation] [Zero] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_5][1], [graylog_5][2], [graylog_5][0], [graylog_5][2], [graylog_5][0]] ...]).
[2018-02-16 16:27:03,665][INFO ][cluster.metadata         ] [Zero] [graylog_5] update_mapping [message]
[2018-02-16 16:27:03,816][INFO ][cluster.metadata         ] [Zero] [graylog_5] update_mapping [message]

Thx

Answer 1

The problem was not elasticsearch indexing but the maximum message size uploaded from rsyslog udp protocol.

To fix it:

/etc/rsyslog.conf

and define

$MaxMessageSize 256k

at the first line.