How to map nested JSON in Log-stash HTTP Output

Question

I am using Logstash to output JSON message to an API. I am using "mapping" attribute to map my message. See, following piece of my shipper configurations.

output {
    stdout { }
     http {
        url => "http://localhost:8087/messages"
        http_method => "post"
        format => "json"
        mapping => ["MessageId","654656","TimeStamp","2001-12-31T12:00:00","CorrelationId","986565","MessageType","%{log_MessageType}" ,"MessageTitle","%{log_MessageTitle}","Message","%{log_Message}"]
    }
}

This configuration is working fine and is producing following output:

{
  "MessageId": "654656",
  "TimeStamp": "2001-12-31T12:00:00",
  "CorrelationId": "986565",
  "MessageType": "INFO",
  "MessageTitle": "TestTittle",
  "Message": "Sample Message"
}

Input Log Entry:

TID: [0] [ESB] [2016-05-30 23:02:02,602]  INFO {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService} -  Configured Registry in 572ms {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService}

Grok Pattern :

TID:%{SPACE}\[%{INT:log_SourceSystemId}\]%{SPACE}\[%{DATA:log_ProcessName}\]%{SPACE}\[%{TIMESTAMP_ISO8601:log_TimeStamp}\]%{SPACE}%{LOGLEVEL:log_MessageType}%{SPACE}{%{JAVACLASS:log_MessageTitle}}%{SPACE}-%{SPACE}%{GREEDYDATA:log_Message}

Problem Statement:

I want following output through mapping of HTTP. I wanted a nested JSON type inside my message, how should I add that in the mapping tag.

Expected Output:

{
  "MessageId": "654656",
  "TimeStamp": "2001-12-31T12:00:00",
  "CorrelationId": "986565",
  "MessageType": "INFO",
  "MessageTitle": "TestTittle",
  "Message": "Sample Message",
  "MessageDetail": {
    "FieldA": "65656",
    "FieldB": "192.168.1.1",
    "FieldC": "sample value"

  }
}

I have tried few options but I am receiving errors.

Answer 1

It is not possible to do this with the message mapping in the http output. That mapping can only create a single-level JSON.

What you can do, however, is to construct your JSON message before it reaches the http output using the mutate/add_field filter.

filter {
   grok {
       match => { "message" => "TID:%{SPACE}\[%{INT:SourceSystemId}\]%{SPACE}\[%{DATA:ProcessName}\]%{SPACE}\[%{TIMESTAMP_ISO8601:log_TimeStamp}\]%{SPACE}%{LOGLEVEL:log_MessageType}%{SPACE}{%{JAVACLASS:log_MessageTitle}}%{SPACE}-%{SPACE}%{GREEDYDATA:log_Message}" }
   }

   # add additional fields in your event here
   mutate {
      gsub => [
        "log_TimeStamp", "\s", "T",
        "log_TimeStamp", ",", "."
      ]
      add_field => {
        "MessageId" => "654656"
        "TimeStamp" => "%{log_TimeStamp}"
        "CorrelationId" => "986565"
        "MessageType" => "%{log_MessageType}"
        "MessageTitle" => "%{log_MessageTitle}"
        "Message" => "%{log_Message}"
        "[MessageDetail][FieldA]" => "65656"
        "[MessageDetail][FieldB]" => "192.168.1.1"
        "[MessageDetail][FieldC]" => "sample value"
      }
      remove_field => ["@version", "@timestamp", "host", "message", "SourceSystemId", "ProcessName", "log_TimeStamp", "log_MessageType", "log_MessageTitle", "log_Message"]
   }
}
output {
   stdout { codec => "rubydebug" }
   http {
      url => "http://localhost:8087/messages"
      http_method => "post"
      format => "json"
   }
}

You'll get exactly the JSON you expect posted to your HTTP endpoint

{
         "MessageId": "654656",
         "TimeStamp": "2016-05-30T23:02:02.602",
     "CorrelationId": "986565",
       "MessageType": "INFO",
      "MessageTitle": "org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService",
           "Message": "Configured Registry in 572ms {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService}",
     "MessageDetail": {
        "FieldA": "65656"
        "FieldB": "192.168.1.1"
        "FieldC": "sample value"
     }
}