ELK Stack on microservices architecture

Question

Hi am trying to use ELK stack in a micro-services architecture which have lots of services distributed over many servers.

Now I have configured Kibana & ElasticSearch. Now my doubt is where do I have to install Logstash. I mean lets say there are three different Servers A, B, & C and all servers have say 3 servcies . Now in total there are 9 services whose logs I want to capture using ELK Stack.

But my question is that do I have to install Logstash on each different server and send parsed logs to Elastic or Can I install Logstash on single server & specify the remote input.

Also do I need to use Filebeat?

Answer 1

It depends. You have two options but in the end you have to install a log forwarder on each server you run your services on.

Logstash

Using only logstash you need to install logstash to each server. The purpose of logstash is to ingest log files and create entries in elasticsearch. Therefore you need to install logstash everywhere you want to ingest log files.

Logstash + Filebeat

Filebeat is a much smaller program whose only purpose is to forward log entries to logstash. It uses less RAM and less CPU load compared to logstash.

In this configuration you need to install filebeat everywhere you need to ingest logs. Then you can centralise logstash on a separate machine where it can parse the log entries without consuming resources on your actual servers.