Logstash indexer is scalable?

Question

I am using logstash shipper -> redis -> logstash indexer -> elastricsearch -> kibana

I understand that redis and elastricsearch are designed to be scalable. How about logstash indexer, I only see on logstash main site show an example of using only single logstash indexer.

Would that be enough to handle all logs? and what is your suggestion on having many instances of logstash indexer?

Answer 1

If your indexer really is the bottleneck of such a setup, you could easily add a second Logstash indexer on a second machine with the same configuration, where you only change the redis source host and the elasticsearch target host. When they read from the same redis instance, they use blocking reads to ensure that one redis entry is read by exactly one indexer.

As with all the components, they are especially good at horizontal scaling (more machines), but the indexer can also be scaled onto a bigger machine.

If the processing of single log events takes too long, you may try to prevent work on the indexer. This could e.g. be done by doing some work on the shipper side (using Logstash filters) or by having the applications write JSON logs instead of "plain" loglines.