Logstash to convert epoch timestamp

Question

I'm trying to parse some epoch timestamps to be something more readable.

I looked around for how to parse them into a normal time, and from what I understand all I should have to do is something like this:

mutate
{
    remove_field => [ "..."]
}

grok
{
    match => { 'message' => '%{NUMBER:time}%{SPACE}%{NUMBER:time2}...' }
}

date
{
    match => [ "time","UNIX" ]
}

An example of a message is: 1410811884.84 1406931111.00 .... The first two values should be UNIX time values.

My grok works, because all of the fields show in Kibana with the expected values, and all the values fields I've removed aren't there so the mutate works too. The date section seems to do nothing.

From what I understand the match => [ "time","UNIX" ] should do what I want (Change the value of time to be a proper date format, and have it show on kibana as a field.) . So apparently I'm not understanding it.

Answer 1

The date{} filter replaces the value of @timestamp with the data provided, so you should see @timestamp with the same value as the [time] field. This is typically useful since there's some delay in the propagation, processing, and storing of the logs, so using the event's own time is preferred.

Since you have more than one date field, you'll want to use the 'target' parameter of the date filter to specify the destination of the parsed date, e.g.:

date {
    match => [ "time","UNIX" ]
    target => "myTime"
}

This would convert the string field named [time] into a date field named [myTime]. Kibana knows how to display date fields, and you can customize that in the kibana settings.

Since you probably don't need both a string a date version of the same data, you can remove the string version as part of the conversion:

date {
    match => [ "time","UNIX" ]
    target => "myTime"
    remove_field => [ "time" ]
}

Answer 2

Consider also trying with UNIX_MS for milliseconds.

date {
    timezone => "UTC"
    match => ["timestamp", "UNIX_MS"]
    target => "@timestamp"
}