Create a logStream for each log file in cloudwatchLogs

Question

I use AWS CloudWatch log agent to push my application log to AWS Cloudwatch.

In the cloudwatchLogs config file inside my EC2 instance, I have this entry:

[/scripts/application]
datetime_format = %Y-%m-%d %H:%M:%S
file = /workingdir/customer/logfiles/*.log
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = /scripts/application

According to this configuration, all log files in workingdir directory are being sent to cloudwatchLogs in the same stream were the name is the instance Id.

My question is, I want for each log file, create a separate logStream, so that the logs reading can be more fast and parseable. In other words, every time I have a new log file, a new logstream is created automatically.

I thought of doing that by a shell script in a cron job but then I'll have to change many other configurations in the architecture, so I'm looking for a way to do it in the config file. In the documentation, they say that :

log_stream_name

Specifies the destination log stream. You can use a literal string or predefined variables ({instance_id}, {hostname}, {ip_address}), or combination of both to define a log stream name. A log stream is created automatically if it doesn't already exist.

The names of the log files can't be 100% predictible, but they always have this structure though:

CustomerName-YYYY-mm-dd.log

Also, another problem is that :

A running agent must be stopped and restarted for configuration changes to take effect.

How can I set the logStream in this case?

Any ideas or suggestions or workarounds are very appreciated.

Answer 1

I know it's been almost two years now, but I wanted to do the exact same thing and couldn't find a way to get it to work. I resorted to the AWS Support, which then confirmed this cannot be done. We're limited to the options offered in the documentation, just like you posted. You can, however, have log groups contain the log file path up to the first dot:

log_group_name – Optional. Specifies what to use as the log group name in CloudWatch Logs. Allowed characters include a-z, A-Z, 0-9, '_' (underscore), '-' (hyphen), '/' (forward slash), and '.' (period).

We recommend that you specify this field to prevent confusion. If you omit this field, the file path up to the final dot is used as the log group name. For example, if the file path is /tmp/TestLogFile.log.2017-07-11-14, the log group name is /tmp/TestLogFile.log.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html