Deleting messages from SQS queue after processing

Question

My application sends messages to a AWS SQS queue for jobs that need some sort of background processing. My processing daemon receives and processes the messages like this:

$result = $sqsClient->receiveMessage([
    'QueueUrl' => \Myapp\Config::get('sqs.queue'),
    'WaitTimeSeconds' => 20,
    'MaxNumberOfMessages' => 1
]);

if (isset($result['Messages'])) {
    foreach ($result->getPath('Messages/*/Body') as $messageBody) {
        $handler = new \Myapp\Handler();
        $handler->dispatch($messageBody);
    }
}

This works fine, however I can see from the SQS console that the messages that my script retrieves are put into the "Messages in Flight" category, and then after a while they're put back into "Messages Available", and my script ends up picking them up again.

How can I remove the messages from the SQS queue, or even better mark them as completed?

Answer 1

When you receive a message from an SQS queue, the message will (by default) return to the queue 30 seconds later. This is to handle cases where processing of the message crashes and the message needs to be processed again.

Once your message is successfully processed, use deleteMessage to delete the message. You'll need the receiptHandle value from the message when you received it from receiveMessage in order to delete the message.

If typical processing of your message may take more than 30 seconds, then you can configure your queue to increase that "return to queue" time. This is called "Default Visibility Timeout" in the SQS queue configuration.

Also be aware that Amazon SQS works in such a way that:

1. messages may be received out-of-order compared to how they were added to the queue
2. messages may be received twice, so allow your message processor to handle these cases

Answer 2

You do need to delete messages after you process them - that would be considered 'marking them completed'; however, if something downstream needs to act on this 'completed'' action, its not uncommon for me to post a 'completion' message to another SQS queue, and then delete the message from the incoming queue. In this way another process, either now or down the road can take action that needs to be done when the first worker completes its work, i.e. chaining those processes together in a decoupled manner.

If there is nothing downstream that needs to be done, simply deleting it is enough.