Menu
I read a blog post about setting up Logstash, Elasticsearch and Kibana, and the author suggested to use NXLog to ship logs from different machines. A typical distributed scenario presented in "The Logstash Book" shows how Logstash can be used in both shipping and indexing role. We are currently experimenting with Logstash and setting up it to ship logs to Elasticsearch was straighforward. So I wonder why people choose NXLog as a log shipper to Logstash instead of using Logstash on both ends.
411
Beats have a small footprint and use fewer system resources than Logstash. Logstash has a larger footprint, but provides a broad array of input, filter, and output plugins for collecting, enriching, and transforming data from a variety of sources.
Filebeat is a lightweight shipper for forwarding and centralizing log data. Installed as an agent on your servers, Filebeat monitors the log files or locations that you specify, collects log events, and forwards them either to Elasticsearch or Logstash for indexing.
Elasticsearch is an open source, full-text search and analysis engine, based on the Apache Lucene search engine. Logstash is a log aggregator that collects data from various input sources, executes different transformations and enhancements and then ships the data to various supported output destinations.
Easily load unstructured data Logstash allows you to easily ingest unstructured data from a variety of data sources including system logs, website logs, and application server logs.
The logstash-forwarder project, formerly known as "Lumberjack", explains it like so:
Resource Usage Concerns
Perceived Problems: Some users view logstash releases as "large" or have a generalized fear of Java.
Actual Problems: Logstash, for right now, runs with a footprint that is not friendly to underprovisioned systems such as EC2 micro instances; on other systems it is fine. This project will exist until that is resolved.
Transport Problems
Few log transport mechanisms provide security, low latency, and reliability.
The lumberjack protocol used by this project exists to provide a network protocol for transmission that is secure, low latency, low resource usage, and reliable.
A Logstash shipper instance isn't especially heavyweight, but if your machine only has 1-2GB of RAM it can be hard to casually allocate several hundred MB to another JVM instance.
Another consideration: what if your shipper nodes are running an OS that isn't supported by Logstash? Logstash can run on Windows, now, but it's still on the buggy side of things. I can't specifically vouch for NXLog in that area, but I gather it's a popular choice.
56
We've implemented shipping with RSyslog rather than LogStash due to a desire to keep java off of our hosts where possible. It makes the LogStash config on the loghost/filer a little more complex, but not having to keep up with Java security upgrades on front-end hosts is more of a pain than dealing with ripping logs apart on the central filer.
21
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
