ELK: How to group stacktraces by exception class in Kibana

Question

I am setting up the ELK stack for a java web application. I successfully parsed multiline java stacktraces with logstash and display the count of exceptions in kibana. Now I would like to display a date histogram with the count of exceptions group by exception class, i.e. 2 java.lang.NullPointerException, 3 java.lang.ArithmeticException per minutes or seconds.

In kibana, I can see the full stacktrace indexed. But I wasn't able to visualize my exceptions group by classes. What is the best practice here? Try to retrieve the fully qualified class name with Logstash and do a term filter in kibana? or is there a way to use the power of ES in kibana?

An example of the beginning of a message field:

2015-08-15 23:23:51.695 [qtp1010279661-1074] ERROR c.m.w.s.proxies.ProxyServlet:71 - Can't get content from url http://localhost:8080/...
org.apache.http.conn.HttpHostConnectException: Connect to localhost:8080 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1, localhost/fe80:0:0:0:0:0:0:1%1] failed: Connection refused
    at org.apache.http.impl.conn.HttpClientConnectionOperator.connect...

My logstash configuration:

input {
    file {
        path => "/Users/dbaq/web-app.log"
        start_position => beginning
    }
}

filter {
    multiline {
        pattern => "%{TIMESTAMP_ISO8601:timestamp}"
        negate => true
        what => "previous"
    }

    grok {
        match => ["message", "(?m)%{TIMESTAMP_ISO8601:timestamp} \[%{DATA:thread}\]\s*%{LOGLEVEL:severity}\s*%{DATA:class}:%{NUMBER:line:int}\s*\- %{GREEDYDATA:message}"]
        overwrite => [ "message" ]
    }

    date {
        match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss.SSS" ]
    }
}

output {
    elasticsearch {
        protocol => "http"
    }
    stdout {}
}

Thanks for your help

EDIT 1: My class field in my logstash pattern represents the class where the exception was thrown, in my example: c.m.w.s.proxies.ProxyServlet. I want to aggregate by my Exception class: org.apache.http.conn.HttpHostConnectException.

Answer 1

As @Alain Collins already pointed out you can use a "data table" for visualization.

I would suggest that you are going to add a multiline codec to your input with the following pattern:

input {
    file {
         path => "/Users/dbaq/web-app.log"
         start_position => beginning
         codec => multiline {
              pattern => "^\s"
              what => "previous"
         }
    }
}

What you can then do is using the predefined grok regex JAVASTACKTRACEPART by doing this:

if "multiline" in [tags] {
    grok {
        match => ["message", "%{JAVASTACKTRACEPART}"]
    }
}

Please note that this will also create a field called class. You can use this field to perform a terms search and get your count metric applied to it.