Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

SNS to Lambda vs SNS to SQS to Lambda

I'm trying to understand whether I need SQS in my workflow if someone can help explain. In my app, when an action is taken, it submits info to SNS topic which invokes Lambda to do some processing. This is working great as it is.

When I do research online, it seems that people are using SQS in this stack as well where SNS would put info on SQS and then SQS would then invoke Lambda.

I guess what I'm trying to understand is the need for SQS in this. What value does that add? In other words, what am I losing by invoking my Lambda directly from SNS?

like image 715
Saumil Shah Avatar asked Mar 07 '17 19:03

Saumil Shah


People also ask

Should I put SQS between SNS and Lambda?

Having SQS in between SNS and Lambda allows reprocessing older unprocessed events in case Lambda fails to process. SQS allows to put a delay, so that message gets processed after some time, it may be useful in the scenario where data takes time to be available.

What is the difference between SNS and SQS?

SQS is mainly used to decouple applications or integrate applications. SNS is used to broadcast messages and it's up to the receivers how they interpret and process those messages.

Why do we need SNS and SQS together?

Using Amazon SNS and Amazon SQS together, messages can be delivered to applications that require immediate notification of an event, and also persisted in an Amazon SQS queue for other applications to process at a later time.

Can SNS send to SQS?

Sending Amazon SNS messages to an Amazon SQS queue in a different account. You can publish a notification to an Amazon SNS topic with one or more subscriptions to Amazon SQS queues in another account.


5 Answers

Primary advantage of having a SQS in between SNS and Lambda is Reprocessing. Assume that the Lambda fails to process certain event for some reason (e.g. timeout or lack of memory footprint), you can increase the timeout (to max 5 minutes) or memory (to max of 1.5GB) and restart your polling and you can reprocess the older events.

This would not be possible in case of SNS to Lambda, wherein if Lambda fails the event is lost. And even if you configure DLQ you would still have to make provisions for reading that separately and processing the message

So if your events are critical and you don't want to miss out on them, then go for SNS - SQS - Lambda

The other advantage of having SQS is cost saving on Lambda invocations (Thanks @codesinthedark for bringing this up). You can have much better scaling and less cost, as it allows you to process messages in batches. So one lambda can be executed for a batch of 10 messages while in case of direct SNS each message would trigger a lambda invocation.

like image 51
Arafat Nalkhande Avatar answered Oct 03 '22 14:10

Arafat Nalkhande


I think couple of things changed in 2019 and SQS can trigger lambda via event source mapping which is mentioned by @alexs. Related blog post: https://aws.amazon.com/about-aws/whats-new/2018/04/aws-lambda-now-supports-amazon-sqs-as-event-source/

To summarize, you can use SQS to lambda with following benefits:

  • Reprocessing events in case of failure and configure how many times a message should be retried before you give up (receive count)
  • Longer retention period
  • Typically chosen in the scenarios where there is a long running job and the lambda polls one by one from job queue.

you can choose to use SNS:

  • If you need to fanout a single message to multiple destinations, say X message should be processed by Y and Z applications. I feel this is the biggest advantage, and if you want reliability in this, you can couple SNS and SQS together.
  • You do not care about lost messages. Remember that there are still retry stratergies when using SNS(linear, geometric, exponential etc)
  • Typically used in the cases where you can ingest/process messages faster. This can sometimes be a problem as well; imagine a scenario where there is a SNS notification for every email that your business receives and you dont have enough lambda concurrency to process all of them. You can solve this by putting an SQS to consume at your own pace.

In both the cases, there can be duplicate messages(in the cases of retry) and there cannot be order guarantees. If you need one, consider Kinesis streams.

like image 23
St1id3r Avatar answered Oct 03 '22 12:10

St1id3r


You can now use SQS as en event source

AWS Lambda Adds Amazon Simple Queue Service to Supported Event Sources

like image 35
alexs Avatar answered Oct 03 '22 14:10

alexs


Adding to @Arafat Nalkhande's answer here are few benefits of SQS's lambda

  1. In SQS we can put a delay, so that message gets processed after some time, it may be useful in the scenario where data takes time to be available.

  2. SQS can serve as a contingency store, lets say downstream services are unavailable, message can be retained in sqs for 15 days.

like image 42
best wishes Avatar answered Oct 03 '22 12:10

best wishes


SQS does not invoke Lambda. SQS cannot invoke anything. People using Lambda with SQS are running Lambda on an event timer, like once a minute, and every time the function runs it polls SQS to see if there is a message to process.

If you don't need to queue things up and prevent too many Lambda functions from running concurrently then you don't need a queue system like SQS.

like image 42
Mark B Avatar answered Oct 03 '22 13:10

Mark B