Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

AWS SQS Long Polling doesn't reduce empty receives

Currently, I have an AWS SQS as a trigger to my AWS Lambda function.

I would like to implement long polling to reduce costs since I've used up 70% of my monthly free tier, mostly from empty receives.

I tried setting up long polling by changing the queue attribute ReceiveMessageWaitTimeSeconds to 20 seconds:

SQS Details showing ReceiveMessageWaitTimeSeconds set to 20s

However, this didn't seem to reduce the number of empty receives, where the settings were changed on 11/19, between 2:00 - 3:00. Empty Receives graph shows the same trend even after setting long polling

According to the AWS Documentation, WaitTimeSeconds has priority over the queue attribute ReceiveMessageWaitTimeSeconds

Short polling occurs when the WaitTimeSeconds parameter of a ReceiveMessage request is set to 0 in one of two ways:

  • The ReceiveMessage call sets WaitTimeSeconds to 0.
  • The ReceiveMessage call doesn’t set WaitTimeSeconds, but the queue attribute ReceiveMessageWaitTimeSeconds is set to 0.

Note

For the WaitTimeSeconds parameter of the ReceiveMessage action, a value set between 1 and 20 has priority over any value set for the queue attribute ReceiveMessageWaitTimeSeconds.

Since AWS Lambda is receiving the SQS requests, I don't think WaitTimeSeconds can be configured.

Why doesn't my long polling configuration work in this situation? Am I misunderstanding something, or did I configure it wrong?

Thank you!

like image 650
naribo Avatar asked Nov 19 '18 09:11

naribo


People also ask

What causes empty receives SQS?

A consumer polling an SQS queue continuously results in empty receives. These empty receives are charged per Amazon SQS pricing even if messages aren't sent or received from your SQS queue.

How does long polling work SQS?

With long polling, the ReceiveMessage request queries all of the servers for messages. Amazon SQS sends a response after it collects at least one available message, up to the maximum number of messages specified in the request. Amazon SQS sends an empty response only if the polling wait time expires.

Which is generally preferred SQS long polling or SQS short polling?

From SQS FAQs: In almost all cases, Amazon SQS long polling is preferable to short polling. Long-polling requests let your queue consumers receive messages as soon as they arrive in your queue while reducing the number of empty ReceiveMessageResponse instances returned.

What is the maximum Amazon SQS long poll time out?

The maximum long polling wait time is 20 seconds. Long polling helps reduce the cost of using Amazon SQS by eliminating the number of empty responses (when there are no messages available for a ReceiveMessage request) and false empty responses (when messages are available but aren't included in a response).


1 Answers

Actually Long Polling is working in your situation.

5 lambdas * polling / 20 seconds * 3600 seconds in an hour = 900 receives/hour

What I think you've missed is the "5 minimum concurrent lambdas". This is implied in the Lambda Scaling Behaviour documentation, but is more helpfully and explicitly laid out in the "Additional Information" section of the announcement/deep-dive blog.

When an SQS event source mapping is initially created and enabled, or when messages first appear after a period with no traffic, then the Lambda service will begin polling the SQS queue using five parallel long-polling connections. The Lambda service monitors the number of inflight messages, and when it detects that this number is trending up, it will increase the polling frequency by 20 ReceiveMessage requests per minute and the function concurrency by 60 calls per minute.

like image 96
thomasmichaelwallace Avatar answered Oct 03 '22 01:10

thomasmichaelwallace