Is it possible to have Centralised Logging for ElasticBeanstalk Docker apps?

Question

We have custom Docker web app running in Elastic Beanstalk Docker container environment. Would like to have application logs be available for viewing outside. Without downloading through instances or AWS console.

So far neither of solutions been acceptable. Maybe someone achieved centralised logging for Elastic Benastalk Dockerized apps?

Solution 1: AWS Console log download

not acceptable - requires to download logs, extract every time. Non real-time.

Solution 2: S3 + Elasticsearch + Fluentd

fluentd does not have plugin to retrieve logs from S3 There's excellent S3 plugin, but it's only for log output to S3. not for input logs from S3.

Solution 3: S3 + Elasticsearch + Logstash

cons: Can only pull all logs from entire bucket or nothing.

The problem lies with Elastic Beanstalk S3 Log storage structure. You cannot specify file name pattern. It's either all logs or nothing. ElasticBeanstalk saves logs on S3 in path containing random instance and environment ids:

s3.bucket/resources/environments/logs/publish/e-<random environment id>/i-<random instance id>/my.log@ 

Logstash s3 plugin can be pointed only to resources/environments/logs/publish/. When you try to point it to environments/logs/publish/*/my.log it does not work. which means you can not pull particular log and tag/type it to be able to find in Elasticsearch. Since AWS saves logs from all your environments and instances in same folder structure, you cannot chose even the instance.

Solution 4: AWS CloudWatch Console log viewer

It is possible to forward your custom logs to CloudWatch console. Do achieve that, put configuration files in .ebextensions path of your app bundle: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.cloudwatchlogs.html

There's a file called cwl-webrequest-metrics.config which allows you to specify log files along with alerts, etc. Great!? except that configuration file format is neither yaml,xml or Json, and it's not documented. There is absolutely zero mentions of that file, it's format either on AWS documentation website or anywhere on the net. And to get one log file appear in CloudWatch is not simply adding a configuration line. The only possible way to get this working seem to be trial and error. Great!? except for every attempt you need to re-deploy your environment.

There's only one reference to how to make this work with custom log: http://qiita.com/kozayupapa/items/2bb7a6b1f17f4e799a22 I have no idea how that person reverse engineered the file format.

cons:

• Cloudwatch does not seem to be able to split logs into columns when displaying, so you can't easily filter by priority, etc.
• AWS Console Log viewer does not have auto-refresh to follow logs.
• Nightmare undocumented configuration file format, no way of testing. Trial and error requires re-deploying whole instance.

Answer 1

Perhaps an AWS Lambda function is applicable?

Write some javascript that dumps all notifications, then see what you can do with those.

After an object is written, you could rename it within the same bucket?

Or notify your own log-management service about the creation of a new object?

Lots of possibilities there...