Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Logstash date filter not updating @timestamp with Apache timestamp

Tags:

I'm trying to backfill some past Apache access log data with logstash, therefore I need the event @timestamp to be set to the date appearing in the log message. This is my current logstash configuration:

input {     tcp {         type => "access_log"         port => 9293     } }  filter {      grok {       match => { "message" => "%{COMBINEDAPACHELOG}" }     }      date {       # Try to pull the timestamp from the 'timestamp' field (parsed above with       # grok). The apache time format looks like: "18/Aug/2011:05:44:34 -0700"       locale => "en"       timezone => "America/New_York"       match => { "timestamp" => "dd/MMM/yyyy:HH:mm:ss Z" }       add_tag => [ "tsmatch" ]     }  }  output {   stdout { codec => rubydebug } } 

However, the date filter doesn't seem to update the event @timestamp, even though the Apache timestamp is catched correctly and the regular expression should match it. The output data looks like this:

{         "message" => "56.116.21.231 - - [20/Nov/2013:22:47:08 -0500] \"GET /xxxx/1.305/xxxx/xxxx.zip HTTP/1.1\" 200 33002333 \"-\" \"xxxxx/3.0.3 CFNetwork/609.1.4 Darwin/13.0.0\"",      "@timestamp" => "2013-12-01T12:54:27.920Z",        "@version" => "1",            "type" => "access_log",            "host" => "0:0:0:0:0:0:0:1%0:51045",        "clientip" => "56.116.21.231",           "ident" => "-",            "auth" => "-",       "timestamp" => "20/Nov/2013:22:47:08 -0500",            "verb" => "GET",         "request" => "/xxxx/1.305/xxxx/xxxx.zip",     "httpversion" => "1.1",        "response" => "200",           "bytes" => "33002333",        "referrer" => "\"-\"",           "agent" => "\"xxxxx/3.0.3 CFNetwork/609.1.4 Darwin/13.0.0\"",            "tags" => [         [0] "tsmatch"     ] } 

Any ideas on what could be wrong?

I'm using the logstash-1.2.2 flatjar.

like image 530
nuqqsa Avatar asked Dec 01 '13 13:12

nuqqsa


1 Answers

Ok, I found the problem, I was using the wrong syntax on the match operation:

match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]

NOT

match => { "timestamp" => "dd/MMM/yyyy:HH:mm:ss Z" }

like image 86
nuqqsa Avatar answered Sep 21 '22 17:09

nuqqsa