AWS Lambda not invoked when subscribed SNS message is large

Question

I've got a AWS lambda function that takes a large array of email addresses and submits them to SES for sending. The function is invoked via a SNS subscription. It works nicely when the message's email array size is small, however when the message's email array size is large the lambda function is NOT invoked. No logging occurs....

I've confirmed that the payload is below the SNS message size required, and I've subscribed to the SNS topic via my email...this works. I get emailed the expected message payload.

Any idea why this fails silently? Suggestions on how to work around this?

Answer 1

For posterity:

Configure the 'Delivery Status' for the SNS topic. It will then log success/failures to CloudWatch.

Expect to see:

167542 byte payload is too large for the Event invocation type (limit 131072 bytes)

So...even though SNS can send a larger payload, that doesn't mean lambda's can be invoked with this size payload. Yuck!

Answer 2

A couple workarounds for this:

1) Post the data to an S3 bucket. Have the Lambda function listen for Object CREATE on that bucket.

2) Post the data to S3, and publish an SNS event with just the bucket and key where the data can be found in S3. Have Lambda subscribe to the SNS topic.

In either case you're going to end up GET-ing the data from an S3 bucket inside your Lambda function. Now you have no size limits. :-)