Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

logstash filter how to create two (or above ) outputs for the one input

Tags:

I am getting via http poller one json

  {
 "id":12345
 "name":"",
 "lastname":"",
 "age":12,
 "address":{"city":"XXXX" , "street":"ZZZZ" }
 }

and I would like this to generate two document in my output :

person :

 {
"id":12345
"name":"",
"lastname":"",
"age":12
  }

address :

 {
"city":"XXXX" , 
"street":"ZZZZ" 
  }

meaning I got one event in the input

in the input phase getting one input :

input {
  http_poller {
    urls => {
      test1 => "http://localhost:8080"
    }
}

in the filter phase I would like to :

  1. create person event (tag it as P)
  2. create address event (tag it as A)

in the output phase I would like to :

  1. send P to P type in ES
  2. send A to A type in ES
like image 260
yoav.str Avatar asked Sep 28 '17 13:09

yoav.str


People also ask

Can Logstash have multiple outputs?

Using Logstash multiple outputs Furthermore, we can forward the filtered data of Logstash either to a single output destination or multiple outputs by filtering the inputs in a specific manner, resulting in the outputs being distributed to that particular stream for each of the inputs received.

Can Logstash have multiple inputs?

Only use input once.

What is mutate filter?

The mutate filter allows you to perform general mutations on fields. You can rename, replace, and modify fields in your events.


1 Answers

You can achieve that with the clone filter.

First, you need to install the plugin which is not bundled by default:

bin/logstash-plugin install logstash-filter-clone

Then you can modify your Logstash config like this:

input {
  http_poller {
    urls => {
      test1 => "http://localhost:8080"
    }
    type => "A"
  }
}
filter {
    clone {
        clones => [ "P" ]
        add_tag => [ "P" ]
    }
    if [type] == "P" {
        mutate {
            remove_field => [ "address" ]
        }
    } else {
        mutate {
            add_tag => [ "A" ]
            remove_field => [ "id", "name", "lastname", "age" ]
        }
    }
}
output {
    elasticsearch {
        hosts => ["localhost:9200"]
        document_type => "%{type}"
    }
}
like image 94
Val Avatar answered Oct 11 '22 12:10

Val