Trying to use Logstash to index FROM Cloudwatch Logs

Question

So we have logs (apache, tomcat, etc) stored in Amazon CloudWatch Logs. I'm trying to use Logstash to index from AWS and send them over to Elasticsearch/Kibana.

I can't seem to find a plugin to accomplish this. Has anyone tried this and was successful? I don't want the metrics, just the logs stored in AWS Logs.

Answer 1

Other posters have mentioned that CloudFormation templates are available that will stream your logs to Amazon Elasticsearch, but if you want to go through Logstash first, this logstash plugin may be of use to you:

https://github.com/lukewaite/logstash-input-cloudwatch-logs/

This plugin allows you to ingest specific CloudWatch Log Groups, or a series of groups that match a prefix into your Logstash pipeline, and work with the data as you will. It is published on RubyGems, and can be installed like a normal Logstash plugin: bin/logstash-plugin install logstash-input-cloudwatch_logs.

Answer 2

As already pointed out by BMW, AWS has just introduced a dedicated CloudWatch Logs Subscription Consumer, which provides one click access to a complete CloudWatch Logs + Elasticsearch + Kibana stack by means of a resp. AWS CloudFormation template, as further illustrated in the introductory blog post.

Given you seem to have an ELK stack readily available, it shouldn't be too complex to adjust the AWS sample template to target your own endpoints instead.