How to use AWS SQS/SNS as a push notification queue for heavy processing tasks via PHP?

Question

I have a single server running on rackspace which is hosting a single PHP web app.

The PHP web app will accept a form submission that then needs to perform a task based on the form field entries.

The task (let's called it the generate metadata task) requires quite a lot of processing time. I was wondering how to allow the form submission to be a straightforward save to database and immediately show success page to user while running the generate metadata task in the background.

I have installed "aws/aws-sdk-php": "~3.11" using composer into the same web app.

My plan is initially this:

code that handles the form submission

$result = $model->save($_POST);
// this code will send the information to either SQS or SNS
$awsClient->sendsMessage($_POST);
if ($result) {
  $this->redirect('success.html');
}

I have read about the fanout architecture stated by AWS.

My issues with the fanout architecture example (as I understand it) are this:

1. the server that sends the message to either SQS or SNS will also be the same server that processes the generate metadata task. In fact, it's the same web app.
2. SQS fulfils the queue part (because I want to execute the tasks in a FIFO and the tasks do take a long time to fulfil). However, it requires my web app to poll SQS continuously. I want a push notification (from AWS to my web app) rather than my web app polling AWS continuously to check for tasks to execute.

I found a possible solution suggested here

The suggested solution is:

1. send the message to a SNS topic.

2. The SNS topic will message both a SQS queue and my web app.

3. My web app, after being triggered, will poll the same SQS queue that has now queued the message continuously until the queue is empty

The drawback I see from this is that my web app will poll the queue before the queue itself has the message.

What is the best way to implement push queues using AWS services?

Answer 1

my web app will poll the queue before the queue itself has the message

You haven't tried it, then, right? :) I'm afraid you're overthinking this. SQS has long polling, which causes the poll request to be suspended at the SQS side until at least one message is available, at which point that message (up to the max number you requested) will be returned. You can set the long poll wait time from 1 to 20 seconds. If no messages are available within this time frame, the response is returned with no messages.

If you poll the queue in response to the notification from SNS, you will find messages there if you use long polling. It's possible for messages to be delayed, but highly unlikely.

The other problem, though, is your assertion that you don't want the app to be constantly polling SQS. I encounter this objection fairly often, and it's often misplaced. With SQS long polling, "constantly" polling an empty queue means one request every 20 seconds. That's 3 req/minute, 180 req/hour, 4320 req/day, 129600 req/month... which turns out to be less than the 1 million free requests allowed each month.

The problem with your server reacting to notifications rather than polling a queue in the background with an appropriate number of workers is that you'll potentially be easily overwhelmed by a large batch of jobs arriving at about the same time. If you get 10 concurrent requests, can you handle it? 100? 1000? Often, for jobs that are asynchronous like this, it costs less (in terms of resources) to request the job than it does to perform the job (e.g., uploading an image should require much less CPU than resizing that image would require). Unless you coordinate your reaction response, you could overwhelm your system.

Don't fall into a conceptual trap of "polling is bad, push is good" where it doesn't apply. The vast majority of the time, that sentiment is absolutely correct... polling is almost always the wrong solution... but with SQS long polling, what you really have is a push mechanism wrapped up in a way that makes it compatible with HTTP, and much of the inherent evil of polling... disappears. If you're in the middle of a long poll, the queue is empty, and a message arrives, your long poll will return with that message almost immediately. It doesn't sit around waiting the timeout to happen. A background process watching the queue may be a good way to go after all.