Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Routing messages from Amazon SNS to SQS with filtering

In RabbitMQ, one can create an exchange, then bind it to multiple queues, each with a routing key. This enables messaging architectures like this:

           message_x
         /    |     \
foo-msg_q  bar-msg_q  msg-logger_q

Clients publish messages to the message_x exchange, which routes only messages with routing key "foo" to the foo-msg_q queue, only messages with the routing key "bar" to the bar-msg_q queue, and all messages to msg-logger_q queue.

I'm having trouble figuring out how to do this in AWS. My first thought was to set up permissions on the individual queues to accept messages based on subject, but the only available fields for permission conditions are:

  • aws:CurrentTime
  • aws:EpochTime
  • aws:MultiFactorAuthAge
  • aws:principaltype
  • aws:SecureTransport
  • aws:SourceArn
  • aws:SourceIp
  • aws:UserAgent
  • aws:userid
  • aws:username

None of these seem like they can be influenced by any message I publish to the message_x topic.

Is it possible to doing something like this when using Amazon Simple Notification Service to fan out to multiple Simple Queue Service queues, with each queue receiving a subset of messages published to the topic?

like image 299
Josh Glover Avatar asked Mar 05 '14 11:03

Josh Glover


People also ask

Can I filter messages in SQS?

When you subscribe an Amazon SQS FIFO queue to an SNS FIFO topic, you can use message filtering to specify that the subscriber receives a subset of messages, rather than all of them. Each subscriber can set its own filter policy as a subscription attribute.

What is filter policy in SNS?

A filter policy is a simple JSON object containing attributes that define which messages the subscriber receives. When you publish a message to a topic, Amazon SNS compares the message attributes to the attributes in the filter policy for each of the topic's subscriptions.

How does SNS and SQS work together?

When you subscribe an Amazon SQS queue to an Amazon SNS topic, you can publish a message to the topic and Amazon SNS sends an Amazon SQS message to the subscribed queue. The Amazon SQS message contains the subject and message that were published to the topic along with metadata about the message in a JSON document.


1 Answers

This is possible by using message attribute filtering in SNS. After you subscribe different SQS queues to an SNS topic, you can specify attributes to filter on by using the SNS API SetSubscriptionAttributes. This will allow messages with different attributes to get routed to the correct SQS queue.

This is also not limited to SQS queues but any subscription sources on a SNS topic. For example, a single SNS topic can publishing one set of messages to Lambda, and another set to SQS.

SDK Reference: http://docs.aws.amazon.com/sns/latest/api/API_SetSubscriptionAttributes.html

More details are given here with examples: https://aws.amazon.com/blogs/compute/simplify-pubsub-messaging-with-amazon-sns-message-filtering/

like image 101
puji Avatar answered Sep 17 '22 15:09

puji