Logstash converting date to valid joda time (@timestamp)

Question

Hope someone can help me out!

I have a question about logstash. I grok the following date with succes: 26/Jun/2013:14:00:26 +0200

Next, I want this date to be used as the @timestamp of the event. As you know logstash automatically adds a timestamp.

Replacing the timestamp that logstash is adding can be done by the date filter. I have added the following date filter: match => [ "date", "dd/MMM/YYYY:HH:mm:ss Z"]

But, for some reason, that doesn't work. When I test it out, I see that logstash just adds his own timestamp.

Code:

grok {
    type => "log-date"
    pattern => "%{HTTPDATE:date}"
}

date{
    type => "log-date"
    match => [ "date", "dd/MMM/YYYY:HH:mm:ss Z"]
}

I need to do this, so I can add events to elasticsearch.

Thanks in advance!

Answer 1

I used the following approach:

# strip the timestamp and force event timestamp to be the same.
# the original string is saved in field %{log_timestamp}.
# the original logstash input timestamp is saved in field %{event_timestamp}.
grok {
  patterns_dir => "./patterns"
  match => [ "message", "%{IRODS_TIMESTAMP:log_timestamp}" ]
  add_tag => "got_syslog_timestamp"
  add_field => [ "event_timestamp", "%{@timestamp}" ]
}

date {
  match => [ "log_timestamp", "MMM dd HH:mm:ss" ]
}

mutate {
        replace => [ "@timestamp", "%{log_timestamp}" ]
}

My problem now is that, even if @timestamp is replaced, I would like to convert it to a ISO8601-compatible format first so that other programs don't have problems interpreting it, like the timestamp present in "event_timestamp":

     "@timestamp" => "Mar  5 14:38:40",
       "@version" => "1",
           "type" => "irods.relog",
           "host" => "ids-dev",
           "path" => "/root/logstash/reLog.2013.03.01",
            "pid" => "5229",
          "level" => "NOTICE",
  "log_timestamp" => "Mar  5 14:38:40",
"event_timestamp" => "2013-09-17 12:20:28 UTC",
           "tags" => [
    [0] "got_syslog_timestamp"
]

You could convert it easily since you have the year information... In my case I would have to parse it out of the "path" (filename) attribute... but still, there does not seem to be an convert_to_iso8901 => @timestamp directive.

Hope this helps with your issue anyway! :)

Answer 2

The above answer is just a work around !, try to add locale => "en" to your code.
If not added, the date weekdays and month names will be parsed with the default platform locale language (spanish, french or whatever) and that's why it didn't work (since your log is in english).

date{
    type => "log-date"
    match => [ "date", "dd/MMM/YYYY:HH:mm:ss Z"]
    locale => "en"
}