Is it possible to fetch data from an HTTP URL to Logstash?

Question

As title says, I need to feed data(CSV or JSON) directly into logstash. What I want is to set a filter, say CSV, which reads the content directly from some http://example.com/csv.php into Logstash without involving any middleman script.

Answer 1

If I understand you correctly, you are trying to call an http resource repeatedly and fetch data into logstash. So you are looking for an input rather than a filter.

Logstash has just released a new http poller input plugin for that purpose. After installing it using bin/plugin install logstash-input-http_poller you can set a config like this to call your resource:

input {
  http_poller {
    urls => {
      myresource => "http://example.com/csv.php"
      }
    }
    request_timeout => 60
    interval => 60
    codec => "json" # set this if the response is json formatted
  }
}

If the response contains CSV you need to set a csv filter.

filter {
    csv{ }
}

There are also plugins which perform an http request within the filter section. However, these are supposed to enrich an existing event and that doesn't seem to be what you are looking for.