Logstash: Received an event that has a different character encoding

Question

when using logstash I see an error like this one :

Received an event that has a different character encoding than you configured. {:text=>"2014-06-22T11:49:57.832631+02:00 10.17.22.37 date=2014-06-22 time=11:49:55 device_id=LM150D9L23000422 log_id=0312318759 type=statistics pri=information session_id=\\\"s617nnE2019973-s617nnE3019973\\\" client_name=\\\"[<IP address>]\\\" dst_ip=\\\"<ip address>\\\" from=\\\"[email protected]\\\" to=\\\"[email protected]\\\" polid=\\\"0:1:1\\\" domain=\\\"machin.fr\\\" subject=\\\"\\xF0\\xCC\\xC1\\xD4\\xC9 \\xD4\\xCF\\xCC\\xD8\\xCB\\xCF \\xDA\\xC1 \\xD0\\xD2\\xCF\\xC4\\xC1\\xD6\\xC9!\\\" mailer=\\\"mta\\\" resolved=\\\"OK\\\" direction=\\\"in\\\" virus=\\\"\\\" disposition=\\\"Quarantine\\\" classifier=\\\"FortiGuard AntiSpam\\\" message_length=\\\"1024\\\"", :expected_charset=>"UTF-8", :level=>:warn}

my logstash.conf is :

 input {
    file{
            path => "/var/log/fortimail.log"
        }

}

 filter  {
    grok {
                    # grok-parsing for logs
        }
}
 output {
    elasticsearch {
            host => "10.0.10.62"
            embedded => true
            cluster => "Mastertest"
            node_name => "MasterNode"
            protocol => "http"
    }
}

I do not know what codec should be used for the correct format of events ?? he problem is in the subject field.

Answer 1

This is because the default charset is UTF-8 and the incoming message contained a character not in the UTF-8 set

To fix this set the charset in the input section using codec and the correct charset. For example

file {
            path => "var/log/http/access_log"
            type => apache_access_log
            codec => plain {
                    charset => "ISO-8859-1"
            }
            stat_interval => 60
}

http://logstash.net/docs/1.3.3/codecs/plain

Answer 2

If u received the logs from external server try to use:

input {
   udp {
     port => yourListenPort
     type => inputType
     codec => plain {
       charset => "ISO-8859-1"
     }
   }
}

i have the same error and i used this, works!