Alert/Notification using Kibana3?

Question

I am using logstash-1.4.1, elasticsearch-1.1.1 and kibana-3.1.0 for analyzing my logs. I am able to view and query my logs.

There's a need in which an alert/notification is needed when a particular log/event happens. Eg: When a Login failed log occurs again and again, an alert/notification (popup, via mail, etc) is required.

Presently I can query my log like for Failed login, but I want it whenever such a log appears a notification/popup appears than me manually querying for it.

Can this be done using the above three ?? How can this be achieved?

Answer 1

Here is how to perform email alerting and monitoring with updated ES and Kibana. I am using elasticsearch-5.5.0, kibana-5.5.0 using XPack and Watcher.

Step 1. Install XPack for Elasticsearch and Kibana

bin/elasticsearch-plugin install x-pack
bin/kibana-plugin install x-pack

Step 2. Restart ES and Kibana

./bin/elasticsearch
./bin/kibana

Step 3. Configure an email account in elasticsearch.yml

xpack.notification.email.account:
    outlook_account: 
        profile: outlook
        email_defaults: 
            from: <sender-email>
        smtp: 
            auth: true
            starttls.enable: true
            host: smtp-mail.outlook.com
            port: 587
            user: <username> 
            password: <password>

** I tried this out with sparkpost, and it worked totally fine. Just changed the profile to sparkpostmail and the host to smtp.sparkpostmail.com. You can follow the guide for other email setups: https://www.elastic.co/guide/en/x-pack/5.6/actions-email.html#configuring-email-actions

Step 4: Configure Email Actions in Kibana Dev Tools (You could do this as a curl command too)

PUT _xpack/watcher/watch/error_report
  {
    "trigger": {
       "schedule": {
          "interval": "1h" <OR TIME INTERVAL TO MONITOR AND ALERT>
                   }
                },
      "input": {
         "search": {
          "request": {
          "indices": [
            "logs"
                      ],
      "body": {
        "query": {
          "match": {
           "message": "error"
                   } 
              }
                }
                 }
                 }
              },
       "actions": {
         "send_email": {
         "email": {
         "to": "<YOUR EMAIL>",
         "subject": "Cluster logs",
         "body": "Cluster Error Logs ",
         "attach_data": true
                   }
                 }
               }
              }

OR! If you want to configure Kibana to send a dashboard or visualization via email, configure the following email action:

PUT _xpack/watcher/watch/error_report
{
    "trigger" : {
    "schedule": {
      "interval": "<TIME_INTERVAL>"
    }
  },
  "actions" : {
    "send_email" : { 
      "email": {
        "to": "<YOUR EMAIL>",
        "subject": "Error Monitoring Dashboard",
        "attachments" : {
            "error_dashboard.pdf" : {
                "reporting" : {
                  "url": "http://<YOUR_HOST>:5601/api/reporting/generate/dashboard/<DASHBOARD_ID>?_g=(time:(from:now-7d%2Fd,mode:quick,to:now))", // This is where you configure settings like time interval
                  "retries":6, 
                  "interval":"15s", 
                  "auth":{ 
                    "basic":{
                       "username":"<USERNAME>",
                       "password":"<PASSWORD>"
                }
              }
            }
          }
        }
      }
    }
  }
}

Step 5 (optional). Delete the watcher when you are finished using Kibana's Dev Tools.

DELETE _xpack/watcher/watch/log_error_watch

This is just a concise update on the above answer for the kibana and xpack updates so it's all in one place! Thanks

Answer 2

There is an email option in logstash in which on detecting a certain pattern in log one can send an email. Look into docs for further reading: http://logstash.net/docs/1.4.1/outputs/email

Answer 3

You can use Watcher for monitoring your Elasticsearch. It alerts you via mail.

For further details, refer to this link:
https://www.elastic.co/products/watcher

You can follow these steps to configure Watcher:

Step 1 – Install Plugin for Watcher (for 1.7):

bin/plugin --install elasticsearch/watcher/latest
bin/plugin --install elasticsearch/license/latest

Step 2 – Restart Elasticsearch:

ES_HOME/bin/elasticsearch

Step 3 – Verify that Watcher is set up:

curl -XGET 'http://localhost:9200/_watcher/stats?pretty'

Step 4 – Watch the log data for errors:

PUT /_watcher/watch/log_error_watch
{
    "trigger": {
        "schedule": {
            "interval": "10m"
        }
    },
    "input": {
        "search": {
            "request": {
                "indices": ["logs"],
                "body": {
                    "query": {
                        "match": {
                            "message": "error"
                        } 
                    }
                }
            }
        }
    },
    "condition": {
        "compare": {
            "ctx.payload.hits.total": {
                "gt": 0
            }
        }
    },
    "actions": {
        "send_email": {
            "email": {
                "to": "<username>@<domainname>",
                "subject": "Cluster logs",
                "body": "Cluster Error Logs ",
                "attach_data": true
            }
        }
    }
}

Step 5 – Configure email (add the lines below into elasticsearch.yml):

watcher.actions.email.service.account:
    work:
    profile: gmail
    email_defaults:
    from: <email> 
    smtp:
    auth: true
    starttls.enable: true
    host: smtp.gmail.com
    port: 587
    user: <username> 
    password: <password> 

Step 6 – To delete Watcher:

curl -XDELETE'http://localhost:9200/_watcher/watch/log_error_watch'