Logstash pattern for nginx error log

Question

This is my sample error log:

2017/03/29 17:32:56 [error] 21924#21924: *212595 access forbidden by rule, client: 172.31.0.14, server: , request: "POST /app/etc/local.xml HTTP/1.1", host: "www.overcart.com"

I want a grok pattern that matches this. I don't know how to proceed. Don't know how to create one. I have tried various but none of them worked.

I am currently parsing this using the following grok pattern:

%{DATESTAMP:mydate} [%{DATA:severity}] (%{NUMBER:pid:int}#%{NUMBER}: *%{NUMBER}|*%{NUMBER}) %{GREEDYDATA:mymessage}(?:, client: (?<client_ip>%{IP}|%{HOSTNAME})) (?:, server: %{IPORHOST:server})(?:, request: %{QS:request})?(?:, host: %{QS:host})?(?:, referrer: \"%{URI:referrer}) 

but it's not parsing from (?:, server: onwards.

Answer 1

Here's the grok pattern I use. It's not perfect and can be improved, but it works and parses also an additional upstream token.

(?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER:threadid}\: \*%{NUMBER:connectionid} %{GREEDYDATA:errormessage}, client: %{IP:client}, server: %{GREEDYDATA:server}, request: "(?<httprequest>%{WORD:httpcommand} %{UNIXPATH:httpfile} HTTP/(?<httpversion>[0-9.]*))"(, )?(upstream: "(?<upstream>[^,]*)")?(, )?(host: "(?<host>[^,]*)")?

Answer 2

this is what I use for nginx-error pattern， it can parse all fields in my error log.:

(?<timestamp>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[%{DATA:err_severity}\] (%{NUMBER:pid:int}#%{NUMBER}: \*%{NUMBER}|\*%{NUMBER}) %{DATA:err_message}(?:, client: (?<clientip>%{IP}|%{HOSTNAME}))(?:, server: %{IPORHOST:server})(?:, request: "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}")?(?:, upstream: "%{DATA:upstream}")?(?:, host: "%{IPORHOST:host}")?(?:, referrer: "%{URI:referrer}”)?

Answer 3

I'm using @mavlarn's grok pattern and it works well for my use case. I'm sure @dr01 will work as well. Remember to test to see what works for you!

Pro-tip: Because there are double quotes inside these grok patterns, in order to make them work properly I needed to surround the pattern with single quotes instead of double.

To highlight, here is an example.

grok {
  match => { "message" => "This "double quote" pattern won't work. Backslashes to cancel/close the regex won't work either. Logstash will fail to start.."}
}
grok {
  match => { "message" => 'But this "double quote" pattern will work when surrounding with single quotes instead.'}
}

Example solution.

grok {
  match => { "message" => '(?<timestamp>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[%{DATA:err_severity}\] (%{NUMBER:pid:int}#%{NUMBER}: \*%{NUMBER}|\*%{NUMBER}) %{DATA:err_message}(?:, client: (?<clientip>%{IP}|%{HOSTNAME}))(?:, server: %{IPORHOST:server})(?:, request: "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}")?(?:, upstream: "%{DATA:upstream}")?(?:, host: "%{IPORHOST:host}")?(?:, referrer: "%{URI:referrer}”)?' }
}

(Tested with Logstash v6.x and v7.x)