rsyslog sending badly encoded (corrupted?) data via tcp (receiving using logstash)

Question

My rsyslog logs locally correctly, however I wanted to also receive the logs remotely, so I added the rule:

*.* @@myIP:5141

to the end of my rsyslog.conf

To receive the output, I'm running logstash with the configuration

input { tcp { port => 5141 } }
output { stdout {} }

Logstash expects UTF-8 encoding, however I get the error

Received an event that has a different character encoding than you configured

The messages themselves seem to be garbled, or a mix of encodings, for example:

\u0016\u0003\u0002\u0000V\u0001\u0000\u0000R\u0003\u0002S\xB1R\xAB5K\xF6\\\xB9\xB2\xB4\xB1\xAE0\t\u007F\xDF`5\xF6\u0015\xC8)H\xD7H\xCF+&\xD5T5\u0000\u0000$\u00003\u0000E\u00009\u0000\x88\u0000\u0016\u00002\u0000D\u00008\u0000\x87\u0000\u0013\u0000f\u0000/\u0000A\u00005\u0000\x84\u0000

Note some entries are \u00, while others are \x. There are even multiple backslashes.

I was wondering if I messed up the settings somehow, or if there is something between me and the server which is messing up the messages?

I have also tried using the syslog logstash input, which gives the same result

Another example:

\u0016\u0003\u0002\u0000V\u0001\u0000\u0000R\u0003\u0002S\xB1RiZ^\xC3\xD9\u001Cj\a\xD4\xE0\xECr\x8E\xAC\xF5\u001A\xB9+\u07B9\xE5\xF9\xA3''z\u0018}9\u0000\u0000$\u00003\u0000E\u00009\u0000\x88\u0000\u0016\u00002\u0000D\u00008\u0000\x87\u0000\u0013\u0000f\u0000/\u0000A\u00005\u0000\x84\u0000

EDIT: I found the source of my problem, and it was encryption related. Unfortunately I can't disclose what I did to fix it, suffice to say John Petrone's answer below is good start for similar problems that future readers may experience

Answer 1

So that magic string you're getting back that looks like broken encoding is actually the SSL Handshake request.

I suspect what you've done is (like I just did) misconfigured the tcp input in logstash. Specifically, I forgot to add the ssl_enable => true. So it was listening for normal TCP and got SSL Handshake and dutifully recorded it as garbage.