logstash(2.3.2) gzip codec not work

Question

I'm using logstash(2.3.2) to read gz file by using gzip_lines codec. The log file example (sample.log) is

127.0.0.2 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"

The command I used to append to a gz file is:

cat sample.log | gzip -c >> s.gz

The logstash.conf is

input { 
  file {
    path => "./logstash-2.3.2/bin/s.gz"
    codec => gzip_lines { charset => "ISO-8859-1"}
  }
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
    #match => { "message" => "message: %{GREEDYDATA}" }
  }
  #date {
  #  match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  #}
}


output {
  stdout { codec => rubydebug }
}

I have installed gzip_line plugin with bin/logstash-plugin install logstash-codec-gzip_lines

start logstash with ./logstash -f logstash.conf

When I feed s.gz with cat sample.log | gzip -c >> s.gz

I expect that the console prints the data. but there is nothing print out.

I have tried it on mac and ubuntu, and get same result. Is anything wrong with my code?

Answer 1

I checked the code for gzip_lines and it seemed obvious to me that this plugin is not working. At least for version 2.3.2. May be it is outdated. Because it does not implement the methods specified here:

https://www.elastic.co/guide/en/logstash/2.3/_how_to_write_a_logstash_codec_plugin.html

So current internal working is like that:

• file input plugin reads file line by line and send it to codec.
• gzip_lines codec tryies to create a new GzipReader object with GzipReader.new(io)
• It then go through the reader line by line to create events.

Because you specify a gzip file, file input plugin tries to read gzip file as a regular file and sends lines to codec. Codec tries to create a GzipReader with that string and it fails.

You can modify it to work like that:

Create a file that contains list of gzip files:

-- list.txt
/path/to/gzip/s.gz 

Give it to file input plugin:

file {
    path => "/path/to/list/list.txt"
    codec => gzip_lines { charset => "ISO-8859-1"}
}

Changes are:

Open vendor/bundle/jruby/1.9/gems/logstash-codec-gzip_lines-2.0.4/lib/logstash/codecs/gzip_lines.r file. Add register method:

public
def register
  @converter = LogStash::Util::Charset.new(@charset)
  @converter.logger = @logger
end

And in method decode change:

@decoder = Zlib::GzipReader.new(data)

as

@decoder = Zlib::GzipReader.open(data)

The disadvantage of this approach is it wont tail your gzip file but the list file. So you will need to create a new gzip file and append it to list.