SQS Logging for insertion and removal from queue

Question

I'm using Amazon SQS for my application in a producer/consumer context. I want to enable queue level logging where I can see items put on the queue and removed from it later. How can I do that?

I have read the following:

http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/logging-using-cloudtrail.html

However, that doesn't suffice for my use case. Are we not allowed to do this with AWS queues?

Answer 1

What you're trying to achieve is not possible with just SQS. Possible solutions include:

1. Implement some middleware API between you producer and SQS queue. API level would log requests from producer.
2. Use Kinesis instead of SQS. Kinesis allows you to replay/analyze records created in last 24 hours.
3. Implement logging in consumer.
4. Use Lambda function that will (with help of CloudWatch Event Rule triggers) read SQS queue once a minute, log records and put them in another SQS queue for later processing by consumer.
5. Use different type of queue that allows logging. For example, Redis has MONITOR command for that.

Answer 2

In addition to Sergey Kovalev answer, one now has the option for Lambda functions to be triggered by SQS events.

You simply:

select the SQS queue you want as the event source for your Lambda function