LogStash: How to make a copy of the @timestamp field while maintaining the same time format?

Question

I would like to create a copy of the @timestamp field such that it uses the same format as @timestamp.

I've tried the following:

mutate
{
    add_field => ["read_time", "%{@timestamp}"]
}

but while @timestamp is in the format: 2014-08-01T18:34:46.824Z,

the read_time is in this format 2014-08-01 18:34:46.824 UTC

This is an issue as Kibana doesn't understand the "UTC" format for histograms.

Is there a way using the date filter to do this?

Answer 1

Kibana can't understand because the read_time field is a string, not a timestamp! You can use ruby filter to do what you need. Just copy the @timestamp to a new field read_time and the field time is in timestamp, not string. The add_field is add a new field with string type!

Here is my config:

input {
    stdin{}
}

filter {
    ruby {
            code => "event['read_time'] = event['@timestamp']"
    }
    mutate
    {
        add_field => ["read_time_string", "%{@timestamp}"]
    }
}

output {
    stdout {
        codec => "rubydebug"
    }
}

You can try and see the output, the output is:

{
   "message" => "3243242",
  "@version" => "1",
"@timestamp" => "2014-08-08T01:09:49.647Z",
      "host" => "BENLIM",
 "read_time" => "2014-08-08T01:09:49.647Z",
"read_time_string" => "2014-08-08 01:09:49 UTC"
}

Hope this can help you.

Answer 2

You don't need to run any Ruby code. You can just use the add_field setting of the Mutate filter plugin:

mutate {
    # Preserve "@timestamp" as "logstash_intake_timestamp"
     add_field => { "logstash_intake_timestamp"=> "%{@timestamp}" }
}
date {
    # Redefines "@timestamp" field from parsed timestamp, rather than its default value (time of ingestion by Logstash)
    # FIXME: include timezone:
    match => [ "timestamp_in_weird_custom_format", "YYYY-MM-dd HH:mm:ss:SSS" ]
    tag_on_failure => ["timestamp_parse_failed"]
    target => "@timestamp"
}