Menu
Bit of background,
I'm using Serverless and .Net to create a lambda with a SQS trigger.
The event trigger is set with batch size of 10k and wait time (Batch Window ie MaximumBatchingWindowInSeconds) of 30 seconds.
Queue's visibility timeout is set to almost 16 minutes.
Now that I've set the lambda to reserved concurrency of only 1 and ran a test where I send 100 items to the queue & was hoping to see only one lambda invocation with exactly those 100 items.
Problem was that it separated the items in the queue and invoked the lambda five times instead, causing five packages to be created as part of the lambda's functionality instead of the one package I wanted. (FYI the lambda's output creates packages in s3 of the messages. I want to have fewer packages that are large.)
Now the question: Is this the expected behavior? and if so why is it so when I've set the queue to accumulate up to 10k items and instead it settled for 15.
According to the aws docs the lambda can grab fewer messages than the batchSize if the payload is larger than 256kb but my messages are very small and 100 messages are no where near 256kb. So that can't be the cause.
Suggestions for alternatives to dealing with this issue are also welcome, right now I'm thinking of running an event bridge scheduler that calls lambda with SQS ReceiveMessage api and creates a single package but then I also have to make sure to properly delete the queue afterwards.
I'm a bit clueless here, I'd appreciate any ideas you guys have. Thanks.
802
With EventBridge (CloudWatch Events), you can create rules that match selected events in the stream and route them to your AWS Lambda function to take action. For example, you can automatically invoke an AWS Lambda function to log the state of an EC2 instance or AutoScaling group.
A Lambda function can process items from multiple queues (using one Lambda event source for each queue). You can use the same queue with multiple Lambda functions.
Yes you can, a single Lambda function can process messages from more than one SQS queue without a problem.
You just created a Lambda function with an SQS event source. You used AWS KMS to encrypt the messages as they passed between Amazon SQS and Lambda. Then you created a function to push messages to the queue to test the event source.
AWS Lambda takes over polling and invoking concurrent Lambda functions to chew through the queue with chunks of messages up to the configured batch size (In our case we configured the batch size to be 10 messages). The Lambda function processes each message in the SQSEvent.
Save the following JSON as a file named input.txt . The preceding JSON simulates an event that Amazon SQS might send to your Lambda function, where "body" contains the actual message from the queue. Run the following invoke AWS CLI command.
To solve this challenge, you decide to use Amazon SQS and Lambda. Amazon SQS is ideal for transmitting a large volume of data, at any throughput, without losing messages or requiring other services to be available. Create an AWS KMS key to encrypt each message body using Amazon SQS server-side encryption (SSE).
Problem was that it separated the items in the queue and invoked the lambda five times instead, causing five packages to be created as part of the lambda's functionality instead of the one package I wanted.
I think that this is probably because there are five SQS pooling threads that lambda uses to pool SQS. From AWS blog:
Lambda service will begin polling the SQS queue using five parallel long-polling connections.
So even though you had reserved concurrency of 1, lambda still uses the 5 threads (you can't control that), and your SQS messages were distributed into these threads. Then, each thread invoked your lambda function, one by one, resulting in observed 5 invocations.
87
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
