Best way to copy messages within 2 SQS queues across AWS accounts

Question

What is the best way to copy messages from one SQS queue to the other across AWS accounts? After searching, I attached the policy granting full read and write access for the IAM user(in the destination account) to the source queue. Then I attached a Permission to the source queue for Everyone to have send and receive message access on the queue.

I am getting below error -

Exception in thread "main" com.amazonaws.AmazonServiceException: The security token included in the request is invalid. (Service: AmazonSQS; Status Code: 403

Probably there is something wrong with the IAM user credentials, however, I have refreshed the credentials, but I'm still getting the error

Answer 1

Permissions are a secondary matter. The primary question is how to "copy messages from one SQS queue to the other across AWS accounts".

Amazon SQS very recently introduced the ability to trigger AWS Lambda functions when a message arrives. You could create a Lambda function that then creates the message in another queue (which could be in a different account, region, etc). However, the original message will be deleted once it is processed, so it isn't really "copying" the message.

The better method would be:

• Instead of sending a message to an Amazon SQS message, send it to an Amazon SNS topic
• You can then subscribe the Amazon SQS queues to the topic -- yes, you can subscribe multiple queues to the topic
• This way, whenever a message is sent to the SNS topic, both queues will receive it

See: Sending Amazon SNS messages to an Amazon SQS queue in a different account - Amazon Simple Notification Service

There is a great video from AWS re:Invent that shows how to use SQS and SNS together.

Update: How to copy existing messages in a queue

"there are existing messages in a queue that needs to be copied to a new queue in a new account."

If that's the case, then you are in trouble!

The idea of a queue is that a message is retrieved processed and deleted. Clearly this is a bad idea for "copying" a message because you don't want to delete them.

You could try increasing the invisibility timeout of the queue, then retrieve all the messages in the queue. They will be placed in-flight, which means they are temporarily invisible but will reappear on the queue if not deleted at the end of the invisibility timeout period. So, your app could read each message (but not delete them) and create new messages in the second queue. Then, each original message would reappear in the original queue.

Or, you could write an app that reads each message and sends it to two queues, deleting the source messages as it goes. Then, treat one of the new queues as a replacement for the original queue.

Bottom line: There is no pre-provided method. You'll have to do it yourself.