how to debug the matching of the date?

Question

I am matching events such as

[Sun Jan 11 10:43:35 2015][3205.51466981] user idp : testing 10.234.22.220 (10.234.22.220) [61673782]

with

%{SYSLOG5424SD:timestamp}%{GREEDYDATA}user %{WORD:user} : testing %{HOST:ip}

It works, I see the various fields in elasticsearch/kibana. Specifically timestamp in the example above is matched with [Sun Jan 11 10:43:35 2015]

I now would like to use this match with date in order to have the right @timestamp.

I tried to use in filter

    date
    {
      match =>  [ "timestamp", "SYSLOG5424SD" ]
    }

but this crashes logstash with an output suggesting to file a bug report - I opened a ticket.

In the meantime I tried to explicitely match the pattern via

    date
    {
      match =>  [ "timestamp", "\[EEE MMM dd HH:mm:ss y\]" ]
    }

As you suspect - it never matches, @timestamp is set to the time when the event is logged by logstash.

Can you spot the problem, or is there a clever way to debug such cases?

Answer 1

The timestamp matching done by the date filter isn't based on regular expressions or grok expressions. That's why putting SYSLOG5424SD there doesn't work. Apart from a couple of special cases listed in the filter documentation you can only use tokens recognized by the Joda-Time library. See the documentation of the joda.time.format.DateTimeFormat class.

You were very close to get it right – just don't escape the square brackets:

date {
  match => ["timestamp", "[EEE MMM dd HH:mm:ss y]"]
}

Again, Joda-Time patterns aren't regular expressions so to match square bracket literals you don't need to do anything special. Quoting the Joda-Time documentation:

Any characters in the pattern that are not in the ranges of ['a'..'z'] and ['A'..'Z'] will be treated as quoted text. For instance, characters like ':', '.', ' ', '#' and '?' will appear in the resulting time text even they are not embraced within single quotes.

Answer 2

Regarding your second question: yes, there is a clever way to debug such cases, there is an online grok debugger (http://grokdebug.herokuapp.com/), and a joda time debugger, I created inspired by the first one: https://java-time-parse-debugger.herokuapp.com/

Update: As users are encouraged to migrate to Java 8's DateTime API, I migrated this debuggung web app to use Java 8's DateTime API, without JodaTime any more.