Using Variables in Elasticsearch Output ILM Rollover Alias

Question

I can't get ILM rollover alias to accept variables. In this particular instance, we'll have an ELK cluster hosting logs for multiple environments. The log entries will be stamped with their environment before coming into the logstash pipeline. I'd like the entries to go to the correct alias, but I get the following error when starting logstash (truncated the stacktrace):

An unexpected error occurred! {:error=>java.net.URISyntaxException: Malformed escape pair at index 0: %{[fields][Environment]}-logs

Here is my logstash pipeline:

input { 
  rabbitmq {
    host => "rabbitmq"
    port => 5672
    user => "guest"
    password => "guest"
    subscription_retry_interval_seconds => 5
    queue => "logstash-queue"
    exchange => "logs"
    exchange_type => "direct"
    durable => true
    key => "logstash"
  }
}

filter {
  mutate {
    rename => {"Properties" => "fields"}
  }
  mutate {
    lowercase => ["[fields][Environment]"]
  }
}

output {
  elasticsearch {
      hosts => ["http://elasticsearch:9200"]
      template_name=>"app-logs"
      ilm_enabled => true
      ilm_rollover_alias => "%{[fields][Environment]}-logs"
      ilm_pattern => "{now/d}-000001"
      ilm_policy => "30_day_retention_logs_policy"
    }
}

Answer 1

According to https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#_writing_to_different_indices_best_practices

You cannot use dynamic variable substitution when ilm_enabled is true and when using ilm_rollover_alias.

The proposed sollution (at the time of writing) is to use multiple outputs. I would suggest automating the logstash.conf, salt, ansible, etc, generation so that you don't have to keep manually editing the config file.

output {
        if <condition> {
            elasticsearch {
                ...
                index => "logstash-<env>-logs"
                ilm...
            }
<etc>

There are issues about this on the elastic github - and it is really strange that this kind of flexibility is removed but there you go.