Regex filtering of messages in SNS

Question

Is there a way to filter messages based on Regex or substring in AWS SNS?

AWS Documentation for filtering messages mentions three types of filtering for strings:

• Exact matching (whitelisting)
• Anything-but matching (blacklisting)
• Prefix matching

I want to filter out messages based on substrings in the messages, for example

I have a S3 event that sends a message to SNS when a new object is added to S3, the contents of the message are as below:

{
    "Records": [
        {
            "s3": {
                "bucket": {
                    "name": "images-bucket"
                },
                "object": {
                    "key": "some-key/more-key/filteringText/additionaldata.png"
                }
            }
        }
    ]
}

I want to keep the messages if only filteringText is present in key field.

Note: The entire message is sent as text by S3 notification service, so Records is not a json object but string.

Answer 1

From what I've seen in the documentation, you can't do regex matches or substrings, but you can match prefixes and create your own attributes in the MessageAttributes field.

To do this, I send the S3 event to a simple Lambda that adds MessageAttributes and then sends to SNS.

In effect, S3 -> Lambda -> SNS -> other consumers (with filtering).

The Lambda can do something like this (where you'll have to programmatically decide when to add the attribute):

let messageAttributes = {
                    myfilterkey: {DataType: "String", StringValue:"filteringText"}
                };
let params = {
                    Message: JSON.stringify(payload),
                    MessageAttributes: messageAttributes,
                    MessageStructure: 'json',
                    TargetArn: SNS_ARN
                };
await sns.publish(params).promise();

Then in SNS you can filter:

{"myfilterkey": ["filtertext"]}

It seems a little convoluted to put the Lambda in there, but I like the idea of being able to plug and unplug consumers from SNS on the fly and use filtering to determine who gets what.