Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Logstash - how do I split an array using the split filter without a target?

I'm trying to split a JSON array into multiple events. Here's a sample input:

{"results" : [{"id": "a1", "name": "hello"}, {"id": "a2", "name": "logstash"}]}

Here's my filter and output config:

filter {
  split {
    field => "results"
  }
}
stdout { 
  codec => "rubydebug"
}

This produces 2 events, one for each of the JSONs in the array. And it's close to what I'm looking for:

{                                              
       "results" => {                          
          "id" => "a1",                        
        "name" => "hello"                      
    },                                         
      "@version" => "1",                       
    "@timestamp" => "2015-05-30T18:33:21.527Z",
          "host" => "laptop",                                      
}                                              
{                                              
       "results" => {                          
          "id" => "a2",                        
        "name" => "logstash"                   
    },                                         
      "@version" => "1",                       
    "@timestamp" => "2015-05-30T18:33:21.527Z",
          "host" => "laptop",                                   
}

The problem is the nested "results" part. "results" being the default value for the target parameter. Is there a way to use the split filter without producing the nested JSON, and get something like this:

{                                                                     
          "id" => "a1",                        
        "name" => "hello"                      
      "@version" => "1",                       
    "@timestamp" => "2015-05-30T18:33:21.527Z",
          "host" => "laptop",                                      
}                                              
{                                              
          "id" => "a2",                        
        "name" => "logstash"                   
      "@version" => "1",                       
    "@timestamp" => "2015-05-30T18:33:21.527Z",
          "host" => "laptop",                                   
}

The purpose is to feed this to the ElasticSearch output with each event being a document with document_id => "id". Any good solutions are welcomed!

like image 607
restassured Avatar asked Oct 20 '22 10:10

restassured


1 Answers

If you know what all of the fields will be (as it appears you do), you can simply rename the fields:

    mutate {
            rename => [
                    "[results][id]", "id",
                    "[results][name]", "name"
            ]
            remove_field => "results"
    }

If you didn't know what all of the fields were, you could write a ruby code filter that did a event['results'].each... and created new fields from the sub-fields of results.

like image 143
Alcanzar Avatar answered Nov 15 '22 10:11

Alcanzar