Azure Service Bus - Two Way Communication Performance Challenge

Question

I need to establish a two-way communication between a Publisher and a Subscriber. This is to facilitate a front-end MVC3 application defining a Subscription with a Correlation Filter, and then placing a message onto a Topic. Finally, the MVC3 controller calls BeginReceive() on the SubscriptionClient, and awaits the response.

The issue seems to be the creation and deletion of these Subscription objects. The overhead is enormous, and it slows the application to a crawl. This is not to mention the various limitations to work around, such as no more than 2000 Subscriptions on a Topic.

What is the best practice for establishing this kind of two-way communication between a Publisher and Subscriber? We want the MVC3 app to publish a message and then wait for a response to that exact message (via the CorrelationId property and a CorrelationFilter). We already cache the NamespaceManager and MessagingFactory, as those are also prohibitively expensive, resource-wise, and also because we were advised that Service Bus uses an explicit provisioning model, where we are expected to pre-create most of these things during role startup.

So, this leaves us with the challenge of correlating request to response, and having this tremendous overhead of the creation and deletion of Subscriptions. What better practice exists? Should we keep a cache of SubscriptionClients, and swap the Filter each time? What does everyone else do? I need to have a request throughput on the order of 5 to 10 thousand MVC3 requests per second through the Web Role cluster. We are already using AsyncController and employing the asynchronous BeginReceive() on SubscriptionClient. It appears to be the creation and deletion of the Subscriptions by the thousands that is choking the system at this point.

UPDATE1: Based on the great advice provided here, we have updated this solution to keep a cache of SubscriptionClient objects on each web role instance. Additionally, we have migrated to a MessageSession oriented approach.

However, this is still not scaling. It seems that AcceptMessageSession() is a very expensive operation. Should MessageSession objects also be cached and re-used? Does each open MessageSession object consume a connection to the Service Bus? If so, is this counted against the Subscription's concurrent connection quota?

Many thanks. I think we are getting there. Most of the example code on the web shows: Create Topic(), then CreateSubscription(), then CreateSubscriptionClient(), then BeginReceive() on the client, then teardown of all of the objects. All I can say is if you did this in real life, your server would be crushed, and you would max out on connections in no time.

We need to put thousands of requests per second through this thing, and it is very apparent that these objects must be cached and reused heavily. So, is MessageSession yet another item to cache? I will have fun caching that, because we will have to implement a reference counting mechanism, where only one reference to the MessageSession can be given out at a time, since this is for http request-specific request/response, and we cannot have other subscribers using the MessageSession objects concurrently.

UPDATE2: OK, it is not feasible to cache MessageSession for re-use, because they only live as long as the LockDuration on the Subscription. This is a bummer, because the maximum LockDuration is 5 minutes. These appear to be for pub/sub of short duration, not for long-running distributed processes. It looks like we need to back to polling Azure Tables.

SUMMARY/COMMENTARY We tried to build on Service Bus because of the scale potential and its durability and delivery semantics. However, it seems that there are situations, high-volume request/response among them, that are not suited to it. The publishing part works great, and having competing consumers on the back-end is great, but having a front-end request wait on a defined, single-consumer response, does not scale well at all, because the MessageSessions take way too long to create via AcceptMessageSession() or BeginAcceptMessageSession(), and because they are not suited to caching.

If someone has an alternative view, I would love to hear it.

Answer 1

This scenario is a classic request/response and a good candidate to use sessions. These are another correlation mechanism. Make a simple request queue and response queue. Each web role thread creates a unique sessionid for a request and puts that value in the "ReplyToSessionID" property of the brokeredmessage. Also this thread calls a AcceptMessageSession on the response queue with the sessionid value so it locks it. The brokered message is sent to the request queue and all worker roles compete for messages. When a worker role gets a request it processes it, creates a response message and sets the sessionid property on the response message = replytosessinid of request. this is then sent to the response queue and will only be delivered to the thread that has locked that session id. A detailed sample using sessions is here. There are 2 additional samples here using Queues and Topics to achieve the request response correlation.

Answer 2

An alternate approach is to perform message correlation on the web role instance by using a dictionary of correlation ids and callback delegates.

Each web role (publisher) instance has a single subscription, filtered by a single subscription id, to the response topic.

Before a message is sent, the callback for that message is registered in the dictionary with the correlation id as the key. The correlation id and a subscription id is sent along with the message such that the response can be sent back to the correct subscription along with the correlation id.

Each web role instance monitors the single subscription and on receiving the response removes the dictionary entry and invokes the callback.