Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

AWS Lambda Triggered by SQS increases SQS request count

I have a AWS Lambda function which is triggered by SQS. This function is triggered approximately 100 times daily, but request count to the SQS queue is approximately 20.000 times daily. I don't understand why the number of requests made to the SQS is too high. My expectation is that the number of requests made to the SQS should be same with the Lambda invocation.

I have only one Lambda function and one SQS queue in my account.

Can be related with polling of SQS queue? I tried to change the polling interval of SQS from the queue configuration but nothing changed. Another possibility is to change polling interval from Lambda function configuration. However, I cannot find any related parameter.

Briefy, I want to reduce number of SQS request, how can i do that while invoking Lmabda function with SQS?

like image 760
Mustafa İlhan Avatar asked Oct 20 '18 10:10

Mustafa İlhan


People also ask

How does Lambda work with SQS trigger?

Simply put, SQS triggers: Trigger a Lambda function when on or when messages have been placed in the queue. Leverage existing retry logic and dead letter queues. If the Lambda function does not return success, the message will not be deleted from the queue and will reappear after the visibility timeout has expired.

How many SQS messages can Lambda process?

Note: When running optimally, Lambda functions with an Amazon SQS queue configured as an event source can scale up to 60 more instances per minute. The maximum number of concurrent invocations is 1,000.

How many requests can SQS handle?

A single Amazon SQS message queue can contain an unlimited number of messages. However, there is a quota of 120,000 for the number of inflight messages for a standard queue and 20,000 for a FIFO queue.

Can a Lambda be triggered by multiple SQS?

A Lambda function can process items from multiple queues (using one Lambda event source for each queue). You can use the same queue with multiple Lambda functions.


1 Answers

When using SQS as an event source for AWS Lambda, AWS Lambda regularly polls the configured SQS queue to fetch new messages. While the official documentation isn't clear really about that, the blog post announcing that feature goes into the details:

When an SQS event source mapping is initially created and enabled, or when messages first appear after a period with no traffic, then the Lambda service will begin polling the SQS queue using five parallel long-polling connections.

According to the AWS documentation, the default duration for a long poll from AWS Lambda to SQS is 20 seconds.

That results in five requests to SQS every 20 seconds for AWS Lambda functions without significant load, which sums up to the ~21600 per day, which is close to the 20000 you're experiencing.

While increasing the long poll duration seems like an easy way to decrease the number of requests, that's not possible, as the 20 seconds AWS Lambda is using by default is already the maximum possible duration for an SQS queue. I'm afraid there is no easy way to decrease the requests to SQS, when using it as event source for AWS Lambda. Instead depending it could be worth evaluating if another event source, like SNS, would fit your use case as well.

like image 102
Dunedan Avatar answered Nov 15 '22 23:11

Dunedan