AWS Lambda missing events from S3 to SQS

Question

I have S3 Create Object -> SQS -> Lambda. In most cases it works (less than 100 new objects). When there are 1K+ objects created in S3, about 10% events do not make it to SQS. We have lambda logging all events as a first step in the function (Java) but these events never show up!

AWS Lambda is configured as Batch Size 10 and Batch Window 10.

Visibility Timeout in both the trigger and sqs is 30 minutes. each event processing in lambda takes 35seconds. Is there any other sqs/trigger/lambda settings I need to configure to not loose events? it doesn't matter if the processing takes many hours.

public String handleRequest(SQSEvent sqsEvent, Context context) {
    LambdaLogger logger = context.getLogger();
    logger.log("\rProcessing Event: " + sqsEvent.toString());
    sqsEvent.getRecords().stream().forEach(
            sqsRecord -> {
                S3EventNotification s3EventNotification = S3EventNotification.parseJson(sqsRecord.getBody());
                logger.log("\rReceived S3EventRecords: " + s3EventNotification.getRecords().size());
                s3EventNotification.getRecords().forEach(
                        s3Record -> {
                            logger.log("\rProcessing s3: " + s3Record.getS3());

Answer 1

This is what AWS says. 'Amazon S3 event notifications are designed to be delivered at least once. Typically, event notifications are delivered in seconds but can sometimes take a minute or longer'. https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html

You may also want to look into the cloudwatch to ensure that all the events have been received in SQS.

SQS Cloudwatch Monitoring

The next step you may want to check is for throttling in Lambda. This also might result in lost events. You may try adjusting the lambda concurrency with a desired value.