Parse JSON message in Logstash

Question

I am sending my jenkins logs to logstash with following config:

 redis {
    host => "localhost"
    key => "logstash"
    data_type => "list"
    codec => json
    }

This works as smooth as expected, now i see follwoing message in KIBANA:

{
  "_index": "logstash-2015.12.18",
  "_type": "logs",
  "_id": "AVG1BN5LXZBIbp7HE4xN",
  "_score": null,
  "_source": {
    "data": {
      "id": "965",
      "projectName": "NicePJ",
      "displayName": "#965",
      "fullDisplayName": "NicePJ",
      "url": "job/NIcePJ/965/",
      "buildHost": "Jenkins",
      "buildLabel": "master",
      "buildNum": 965,
      "buildDuration": 1,
      "rootProjectName": "NicePJ",
      "rootProjectDisplayName": "#965",
      "rootBuildNum": 965,
      "buildVariables": {
        "target_SUT": "0201",
        "report_warnings": "false",
        "product": "Ours",
        "testsuite": "Exciting_stuff5",
        "qft_version": "current",
        "target_task": "t324",
        "branch": "test",
        "testcase": "",
        "revision": "HEAD",
        "node": "hsqs960",
        "client": "Desktop",
        "run_specific_test": "false",
        "user": "xxxxx"
      }
    },
    "message": [
      "A         This is a message XYZ"
    ],
    "source": "jenkins",
    "source_host": "http://serverXL:8080/",
    "@timestamp": "2015-12-18T12:16:02.000Z",
    "@version": 1
  },
  "fields": {
    "@timestamp": [
      1450440962000
    ]
  },
  "sort": [
    1450440962000
  ]
}

Now i want to filter the message field for certain messages, but i cant get it work. How can i filter the message field and how can i access the buildHost field to use it in an if statement in the pipeline?

Following i tried after many examples:

 if[data][buildHost]== "jenkins"
  {
         grok
         {
           match => { "message[0]"  => "\[exec\]\s*\<%{GREEDYDATA:test}\s*\[%{GREEDYDATA:result}\]" }
         }
  }

But this is just not working at all, please help me out.

Answer 1

Conditional

The == compares simple string and case sensitive, so "jenkins" will not match as your data shows ("buildHost": "Jenkins",):

if[data][buildHost]== "jenkins"

But following does:

if[data][buildHost]== "Jenkins"

If you need match both, you can either use || or regex =~.

Grok

The grok is a filter to parse message with regex pattern. You can test your regex pattern with

• online grok debugger
• Kibana dev tools's grok debugger