subscription filter for AWS CloudWatch logs to weed out Lambda Report messages

Question

I'm using a subscription filter to get logs from a specific log group to Firehose which will eventually put it into Elasticsearch. The logs in this log group are from a Java Lambda. All the START RequestId ..., END RequestId ... and REPORT RequestId ... messages also end up in Elasticsearch.

Is it possible to have a subscription filter so that these messages don't reach firehose and only the actual log messages from Lambda function reach the firehose. Or, is processing them with a "Transformation Lambda" the only way to achieve this ?

Answer 1

Yes it's possible. It's a bit cryptic, but here's a filter pattern that will do the trick:

[a != "START" && a != "END" && a != "REPORT" && a != "RequestId:", ...]

When tested against:

START RequestId: 9538d388-c156-4680-b9d0-ba98c73742c7 Version: $LATEST
2019-02-06T20:30:49.096Z    9538d388-c156-4680-b9d0-ba98c73742c7    Hello World
END RequestId: 9538d388-c156-4680-b9d0-ba98c73742c7
REPORT RequestId: 9538d388-c156-4680-b9d0-ba98c73742c7  Duration: 24.45 ms  Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 47 MB  
RequestId: 9538d388-c156-4680-b9d0-ba98c73742c7 Process exited before completing request

Only this will match:

2019-02-06T20:30:49.096Z    9538d388-c156-4680-b9d0-ba98c73742c7    Hello World