dynamic URL parameters with logstash http-poller

Question

I would like to use logstash's http-poller input to consume a webservice returning an array of json documents to load data into elasticsearch. In order to do this, the webservice GET URL needs to be updated with URL query parameters between requests.

For example, the web service takes a date range specified as start and end parameters and limits responses to 1000 results. When first loading data, I would like logstash to continually call the web service with the start time getting set to the latest timestamp from the previous request until all data has been loaded.

To clarify, the first HTTP GET request should look like webserviceurl?start=2015-01-01T00:00:00. Assume the latest document in the response has the timestamp 2015-01-01T21:55:34. Then the next HTTP GET request should look like webserviceurl?start=2015-01-01T21:55:34

Is this possible with logstash and the http-poller input?

Answer 1

This is not currently possible, but this issue requests the enhancement.

Currently, http_poller among many other Logstash input plugins have no notion of state.

As explained in Event Dependent Configuration:

Some of the configuration options in Logstash require the existence of fields in order to function. Because inputs generate events, there are no fields to evaluate within the input block—they do not exist yet!