Forwarding journald to Cloudwatch Logs

Question

I'm a newbie to CentOS and wanted to know the best way to parse journal logs to CloudWatch Logs.

My thought processes so far are:

• Use FIFO to parse the journal logs and ingest this to Cloudwatch Logs, - It looks like this could come with draw backs where logs could be dropped if we hit buffering limits.

• Forward journal logs to syslog and send syslogs to Cloudwatch Logs --

The idea is essentially to have everything logging to journald as JSON and then forward this across to CloudWatch Logs.

What is the best way to do this? How have others solved this problem?

Answer 1

Take a look at https://github.com/advantageous/systemd-cloud-watch

We had problems with journald-cloudwatch-logs. It just did not work for us at all.

It does not limit the size of the message or commandLine that it sends to CloudWatch and the CloudWatch sends back an error that journald-cloudwatch-logs cannot handle which makes it out of sync.

systemd-cloud-watch is stateless and it asks CloudWatch where it left off.

systemd-cloud-watch also creates the log-group if missing.

systemd-cloud-watch also uses the name tag and the private ip address so that you can easily find the log you are looking for.

We also include a packer file to show you how to build and configure a systemd-cloud-watch image with EC2/Centos/Systemd. There is no question about how to configure systemd because we have a working example.

Answer 2

Take a look at https://github.com/saymedia/journald-cloudwatch-logs by Matin Atkins.

This open source project creates a binary that does exactly what you want - ship your (systemd) journald logs to AWS CloudWatch Logs.

The project depends on libsystemd to forward directly to CloudWatch. It does not rely on forwarding to syslog. This is a good thing.

The project appears to use golang's concurrent channels to read the logs and batches writes.