How to correctly parse text file using rsyslog and imfile

Question

Good day

I want to import text files into rsyslog, using the imfile file input module. However, rsyslog does not parse the content of the text files as I expected and I am struggling to find documentation on exactly how it is done. To test the setup, I am reading from a text file with imfile and then writing the logs to another text file with omfile.

The text file's contents are logs in the "standard" syslog format:

<PRI>TIMESTAMP HOSTNAME MESSAGE

The example text file (example_file.txt.) that I want to import into Rsyslog looks like this:

<34>Feb 15 12:12:12 hostname1 tag1: message1
<34>Feb 16 12:12:12 hostname2 tag2: message2
<34>Feb 17 12:12:12 hostname3 tag3: message3

My config file for rsyslog in rsyslog-d looks like follows:

module(load = "imfile")
input(type = "imfile" file = "/home/.../Desktop/example_file.txt" Tag = "example")
action(type = "omfile" file = "/home/.../Desktop/example_output.log")

The resulting output in example_output.log looks like this:

Feb 15 17:10:21 username example <34>Feb 15 12:12:12 hostname1 tag1: message1
Feb 15 17:10:21 username example <34>Feb 16 12:12:12 hostname1 tag2: message2
Feb 15 17:10:21 username example <34>Feb 17 12:12:12 hostname1 tag3: message3

As you can see, all of the content from example_file.txt was placed in the MSG field of the resulting log in example_output.log, instead of using the field information and placing them in the correct places, e.g. TIMESTAMP, HOSTNAME, TAG, MSG. I have played around with different formats in the .txt file, or even saving the .txt file as a .log file, but rsyslog places the whole content in the MSG field every time.

My question then:

How can I tell rsyslog and imfile that my .txt content is actually logs and to parse them correctly?

Take into account:

1. I am working on the Up-Board with Linux v4.4.0-ubi4-amd64 (UbiLinux)

2. I am using rsyslog8.24 (newest stable version)

3. I have already read through:

   -Rsyslog official documentation,

   -Imfile official documentation,

   -Rainer Gerhards's syslog parsing in rsyslog (http://www.rsyslog.com/doc/syslog_parsing.html),

   -and even the documentation for the BSD Syslog protocol RFC3164 (http://www.ietf.org/rfc/rfc3164.txt)

Answer 1

You can use templates to extracts fields out of messages. Here is an example template.

template(name="structured-format" type="list") {
   constant(value="{")
       property(outname="pri" name="msg" field.number="1" field.delimiter="32" format="jsonf") 
       constant(value=", ")
       property(outname="hostname" name="msg" field.number="4" field.delimiter="32" format="jsonf") 
       constant(value=", ")
       property(name="msg" format="jsonf")
   constant(value="} \n")
}

You can use this template in output like this.

action(type = "omfile" file = "/home/.../Desktop/example_output.log" template="structured-format") 

Output would look like this:

{"pri":"<34>", "hostname":"hostname1", "msg":"<34>Feb 15 12:12:12 hostname1 tag1: message1"}

That said, I have not yet figured out how to exclude parsed fields from msg and only add the remaining to msg field. Hope you find the pointers helpful.