logstash parsing timestamp halfday am/pm

Question

New to logstash, really enjoying it.

Trying to parse a CSV file containing a timestamp. Would like to parse the timestamp and use it as the @timestamp field.

Sample of my CSV input

input {
    stdin {}
}

filter {
    # filter the input by csv (i.e. comma-separated-value)
    csv {
        columns => [
            "Job ID",
            "Server Name",
            "Status Code",
            "Job Type",
            "Client Name",
            "Start Time",
            "End Time"
        ]
    }
    # parse the start time to create a real date
    date {
        # Examples of times in this log file
        # "May 29, 2015 10:00:01 PM"
        # "May 9, 2015 4:47:23 AM"
        match => [ "End Time",
                   "MMM dd, YYYY HH:mm:ss aa",
                   "MMM  d, YYYY HH:mm:ss aa" ]
    }
}

# send the output to stdout, using the rubydebug codec
# rubydedug uses the Ruby Awesome Print library
output {
    stdout { codec => rubydebug }
}

Sample of my input

108628,anmuswcnbu01,1,Backup,anmuswcrfax01.na.jnj.com,"May 29, 2015 10:00:01 PM","May 30, 2015 6:21:29 AM"
108629,anmuswcnbu01,1,Backup,anmuswcapps01.na.jnj.com,"May 29, 2015 10:00:01 PM","May 9, 2015 10:51:39 pm"
108630,anmuswcnbu01,1,Backup,anmuswcapps03.na.jnj.com,"May 29, 2015 10:00:01 PM","May 29, 2015 9:31:19 PM"

Sample of my output

Logstash startup completed
{
        "message" => [
        [0] "108628,anmuswcnbu01,1,Backup,anmuswcrfax01.na.jnj.com,\"May 29, 2015 10:00:01 PM\",\"May 30, 2015 6:21:29 AM\"\r"
    ],
       "@version" => "1",
     "@timestamp" => "2015-05-30T06:21:29.000Z",
           "host" => "ip-172-31-34-14",
         "Job ID" => "108628",
    "Server Name" => "anmuswcnbu01",
    "Status Code" => "1",
       "Job Type" => "Backup",
    "Client Name" => "anmuswcrfax01.na.jnj.com",
     "Start Time" => "May 29, 2015 10:00:01 PM",
       "End Time" => "May 30, 2015 6:21:29 AM"
}
{
        "message" => [
        [0] "108629,anmuswcnbu01,1,Backup,anmuswcapps01.na.jnj.com,\"May 29, 2015 10:00:01 PM\",\"May 9, 2015 10:51:39 pm\"\r"
    ],
       "@version" => "1",
     "@timestamp" => "2015-05-09T10:51:39.000Z",
           "host" => "ip-172-31-34-14",
         "Job ID" => "108629",
    "Server Name" => "anmuswcnbu01",
    "Status Code" => "1",
       "Job Type" => "Backup",
    "Client Name" => "anmuswcapps01.na.jnj.com",
     "Start Time" => "May 29, 2015 10:00:01 PM",
       "End Time" => "May 9, 2015 10:51:39 pm"
}
{
        "message" => [
        [0] "108630,anmuswcnbu01,1,Backup,anmuswcapps03.na.jnj.com,\"May 29, 2015 10:00:01 PM\",\"May 29, 2015 9:31:19 PM\"\r"
    ],
       "@version" => "1",
     "@timestamp" => "2015-05-29T09:31:19.000Z",
           "host" => "ip-172-31-34-14",
         "Job ID" => "108630",
    "Server Name" => "anmuswcnbu01",
    "Status Code" => "1",
       "Job Type" => "Backup",
    "Client Name" => "anmuswcapps03.na.jnj.com",
     "Start Time" => "May 29, 2015 10:00:01 PM",
       "End Time" => "May 29, 2015 9:31:19 PM"
}
Logstash shutdown completed

For example, in that last (3rd) row of data, rather than:

"@timestamp" => "2015-05-29T09:31:19.000Z",

I feel like I should be getting

"@timestamp" => "2015-05-29T21:31:19.000Z",

As best I can tell, the date filter seems to be ignoring my "half-day" syntax

 match => [ "End Time",
                   "MMM dd, YYYY HH:mm:ss aa",
                   "MMM  d, YYYY HH:mm:ss aa" ]

New to logstash, so wondering if I'm doing something wrong ?

-Chad

Answer 1

The date filter uses a format compatible with Joda-Time.

Quoting part of Joda's symbol table:

 Symbol  Meaning                      Presentation  Examples
 ------  -------                      ------------  -------   
 a       halfday of day               text          PM
 K       hour of halfday (0~11)       number        0
 h       clockhour of halfday (1~12)  number        12

 H       hour of day (0~23)           number        0
 k       clockhour of day (1~24)      number        24
 m       minute of hour               number        30
 s       second of minute             number        55
 S       fraction of second           number        978

It's easy to overlook, but halfday hours in your case are KK instead of HH.