How to setup ELK with node.js

Question

I want to log error from my node.js server to another server. I'm thinking of using elasticsearch, logstash and kibana. I want to know how to setup ELK with my node server.

Answer 1

I had exactly this use case in my older organization. A basic tutorial to startup with Beats + ELK - https://www.elastic.co/guide/en/beats/libbeat/current/getting-started.html

So basically this is how it works - Your nodejs app will log in the files (you can use bunyan for this) in different formats like error/warning/info etc. Filebeat will tail these log files and send messages to logstash server. Logstash input.conf will have some input filters (in your case it will be error filters). When any log message passes these filters then logstash will forward it to some endpoint as decided in output.conf file.

Here is what we did -

Initial architecture - Install filebeat (earlier we used logstash forwarder) client to tail the logs on nodejs server and forward it to logstash machine. Logstash will do some processing on input logs and send them to ES cluster (can be on same machine as Logstash). Kibana is just a visualization on this ES.

Final Architecture - Initial setup was cool for small traffic but we realized that logstash can be single point of failure and may result in log loss when traffic increased. So we integrated Kafka along with Logstash so that system scales smoothly. Here is an article - https://www.elastic.co/blog/logstash-kafka-intro

Hope this helps!

Answer 2

You can install Logstash in the NodeJS Server, and then create a configuration file that accepts input (location of the log file(s)) and output to your Elastic Server host.

Below is the sample configuration file (custom.conf) which has to created in your logstash directory.

    input {
    file {
    path => "/path to log"
    start_position => beginning
    }
    }
    output {
    stdout { }
    elasticsearch{
    type => "stdin-type"
    embedded => false
    host => "192.168.0.23"
    port => "9300"
    cluster => "logstash-cluster"
    node_name => "logstash"
    }
    }

Execute the logstash

    logstash -f custom.conf

Reference: https://www.elastic.co/guide/en/logstash/current/config-examples.html

If you are planning to customize a NodeJS application for sending error logs then you can install some ELKStack Nodjs wrapper and post error log within your application. ELKStack Nodjs wrapper - https://www.npmjs.com/package/elksdk