Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Suggestion on Logstash Setup

Tags:

logstash

I've implemented logstash ( in testing ) as below mentioned architecture.

enter image description here

Component Break Down

  • Rsyslog client: By default syslog installed in all Linux destros, we just need to configure rsyslog to send logs to remote server.
  • Logstash: Logstash will received logs from syslog client and it will store in Redis.
  • Redis: Redis will work as broker, broker is to hold log data sent by agents before logstash indexes it. Having a broker will enhance performance of the logstash server, Redis acts like a buffer for log data, till logstash indexes it and stores it. As it is in RAM its too fast.
  • Logstash: yes, two instance of logstash, 1st one for syslog server, 2nd for read data from redis and send out to elasticsearch.
  • Elasticsearch: The main objective of a central log server is to collect all logs at one place, plus it should provide some meaningful data for analysis. Like you should be able to search all log data for your particular application at a specified time period.Hence there must be a searching and well indexing capability on our logstash server. To achieve this, we will install another opensource tool called as elasticsearch.Elasticsearch uses a mechanism of making an index, and then search that index to make it faster. Its a kind of search engine for text data.
  • Kibana : Kibana is a user friendly way to view, search and visualize your log data

But I'm little bit confuse with redis. using this scenario I'll be running 3 java process on Logstash server and one redis, this will take hugh ram.

Question Can I use only one logstash and elastic search ? Or what would be the best way ?

like image 825
Rahul Patil Avatar asked Jun 04 '14 10:06

Rahul Patil


People also ask

How can I improve my Logstash performance?

Solution for improving overall performance This can be accomplished by running multiple (identical) Logstash pipelines in parallel within a single Logstash process, and then load balancing the input data stream across the pipelines.

What is a benefit of Logstash?

Logstash benefits Logstash allows you to easily ingest unstructured data from a variety of data sources including system logs, website logs, and application server logs.

What is Logstash and how is it useful in an elk stack?

Logstash is a server‑side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a "stash" like Elasticsearch. Kibana lets users visualize data with charts and graphs in Elasticsearch. The Elastic Stack is the next evolution of the ELK Stack.


1 Answers

I am actually in the midst of setting up logstash, redis, elasticsearch, kibana (aka ELK architecture) at my company.

I have the processes split between virtual machines. While you could put them on the same machine, what happens if a machine dies? Then you are left with your indexer and cluster down at the same time.

You also have the problem of not being able to properly replicate your shards on the Elasticsearch. Since you only have one server, the shards won't be replicated and your cluster health will always be yellow. You need to add enough servers to avoid the split-brain scenario.

Why keep Redis?

Since Redis can talk to multiple logstash indexers, one key point is that this makes the indexing transparent to your shippers in that if one indexer goes down, the alternates will pick up the load. This makes your setup high availability.

It's not just a matter of shipping logs and having them indexed and searchable. While your setup will likely work in a very small, rare situation, the stuff people are doing with ELK setups are hundreds of servers, even thousands, so the ELK architecture is meant to scale. All of these servers will also need to be remotely managed by something called Puppet.

Finally, if you have not read it yet, I suggest you read The Logstash Book by James Turnbull.

The following are some more recommended books that have helped me so far:

  • Pro Puppet, Second Edition
  • Elasticsearch Cookbook, Second Edition
  • Redis Cookbook
  • Redis in Action
  • Mastering Elasticsearch
  • ElasticSearch Server
  • Elasticsearch: The Definitive Guide
  • Puppet Types and Providers
  • Puppet 3 Cookbook
like image 126
Engineer2021 Avatar answered Sep 19 '22 10:09

Engineer2021