Azure Service Bus - Round Robin Topic to Multiple Services

Question

Here's the scenario:

Publisher #1 ═══╗             ╔═══ Round Robin ═══╦═══ Subscriber #1 (Service 1)
                ║             ║                   ╚═══ Subscriber #2 (Service 1)
                ╠═══ Topic ═══╣
                ║             ║                   ╔═══ Subscriber #3 (Service 2)
Publisher #2 ═══╝             ╚═══ Round Robin ═══╩═══ Subscriber #4 (Service 2)

I have one message that needs to be processed by multiple subscribers, BUT only one per service (There will be multiple instances running for each service).

Message #1, needs to be processed by Subscribers #1 and #3. Message #2, needs to be processed by Subscribers #2 and #4. Message #3, subscribers #1 and #3 again. Basically, each message should round robin to each of the load balanced services that are subscribing to each message, grouped by each service that is connecting. Is this possible without creating multiple topics?

Even if it's not round robin per-say, I'd like to use best effort to load balance across multiple services. Is this possible?

Answer 1

1. Topics

Topics are a publishing/distribution mechanism that will send the message once per subscription (subscriber).

A topic subscription resembles a virtual queue that receives copies of the messages that are sent to the topic. Messages are received from a subscription identically to the way they are received from a queue...

Subscriptions support the same patterns described earlier in this section with regard to queues: competing consumer, temporal decoupling, load leveling, and load balancing.

Source: MSDN Article

You need to re-use the topic subscription among competing consumers (service instances) to achieve your scenario.

Publisher #1 ═══╗             ╔═══ Subscription 1 ═══╦═══ Service 1-instance 1
                ║             ║                      ╚═══ Service 1-instance 2
                ╠═══ Topic ═══╣
                ║             ║                      ╔═══ Service 2-instance 1
Publisher #2 ═══╝             ╚═══ Subscription 2 ═══╩═══ Service 2-instance 2

A. Creating a Topic subscription

string connectionString = "<Secret>"
var namespaceManager =
    NamespaceManager.CreateFromConnectionString(connectionString);

if (!namespaceManager.SubscriptionExists("TestTopic", "Inventory"))
{
    namespaceManager.CreateSubscription("TestTopic", "Inventory");
}

B. Listen to an existing subscription

MessagingFactory factory = MessagingFactory.Create(uri, tokenProvider);

MessageReceiver receiver = factory.CreateMessageReceiver("TestTopic/subscriptions/Inventory");

2. Queues

Using multiple Queues could also fit your specific scenario. Each Queue having multiple competing consumers (instances) will deliver the message only once to the first client that request and process it successfully.

The design then becomes:

Publisher #1 ═══╗         ╔═══ Service 1 Queue ═══╦═══ Subscriber #1 (Service 1)
                ║         ║                       ╚═══ Subscriber #2 (Service 1)
                ╠═══ASB═══╣
                ║         ║                       ╔═══ Subscriber #3 (Service 2)
Publisher #2 ═══╝         ╚═══ Service 2 Queue ═══╩═══ Subscriber #4 (Service 2)

Answer 2

I'd like to use best effort to load balance across multiple services. Is this possible?

From your description, it seems you're trying to load balance between multiple instances of any given service, not a a single service. You get that out of the box with competing consumer pattern that ASB supports. Not really something you need to work for.

The problem I'm trying to solve, is I only want 1 consumer in each service consuming messages on a single topic. Is this possible with ASB?

Yes, that is possible. If you have a shared topic with subscription per service (not instance) and containing a rule that always evaluates to true, SqlFilter (1 = 1). Then each subscriber will get a copy of that message. In essence you'll be broadcasting the message to all of your services. Thanks to competing consumer, only one instance of each service will get that message.

To target a specific service, you'd create an additional rule that would filter out message on a property (a standard property or a custom one). For example, it could be a service name.

If you don't want to focus on the the middleware for your microservices, you can take a peek at frameworks that do it for you. Such frameworks usually tend to provide additional features as well. Have a look at NServiceBus or MassTransit for example.

Full disclaimer - I'm working on NServiceBus and its ASB transport.