Azure Service Bus Subscriber regularly phoning home?

Question

We have pub/sub application that involves an external client subscribing to a Web Role publisher via an Azure Service Bus Topic. Our current billing cycle indicates we've sent/received >25K messages, while our dashboard indicates we've sent <100. We're investigating our implementation and checking our assumptions in order to understand the disparity.

As part of our investigation we've gathered wireshark captures of client<=>service bus traffic on the client machine. We've noticed a regular pattern of communication that we haven't seen documented and would like to better understand. The following exchange occurs once every 50s when there is otherwise no activity on the bus:

1. The client pushes ~200B to the service bus.
2. 10s later, the service bus pushes ~800B to the client. The client registers the receipt of an empty message (determined via breakpoint.)
3. The client immediately responds by pushing ~1000B to the service bus.

Some relevant information:

• This occurs when our web role is not actively pushing data to the service bus.
• Upon receiving a legit message from the Web Role, the pattern described above will not occur again until a full 50s has passed.
• Both client and server connect to sb://namespace.servicebus.windows.net via TCP.
• Our application messages are <64 KB

Questions

1. What is responsible for the regular, 3-packet message exchange we're seeing? Is it some sort of keep-alive?
2. Do each of the 3 packets count as a separately billable message?
3. Is this behavior configurable or otherwise documented?

EDIT:

This is the code the receives the messages:

    private void Listen()
    {
        _subscriptionClient.ReceiveAsync().ContinueWith(MessageReceived);
    }

    private void MessageReceived(Task<BrokeredMessage> task)
    {
        if (task.Status != TaskStatus.Faulted && task.Result != null)
        {
            task.Result.CompleteAsync();
            // Do some things...
        }
        Listen();
    }

Answer 1

I think what you are seeing is the Receive call in the background. Behind the scenes the Receive calls are all using long polling. Which means they call out to the Service Bus endpoint and ask for a message. The Service Bus service gets that request and if it has a message it will return it immediately. If it doesn't have a message it will hold the connection open for a time period in case a message arrives. If a message arrives within that time frame it will be returned to the client. If a message is not available by the end of the time frame a response is sent to the client indicating that no message was there (aka, your null BrokeredMessage). If you call Receive with no overloads (like you've done here) it will immediately make another request. This loop continues to happend until a message is received.

Thus, what you are seeing are the number of times the client requests a message but there isn't one there. The long polling makes it nicer than what the Windows Azure Storage Queues have because they will just immediately return a null result if there is no message. For both technologies it is common to implement an exponential back off for requests. There are lots of examples out there of how to do this. This cuts back on how often you need to go check the queue and can reduce your transaction count.

To answer your questions:

1. Yes, this is normal expected behaviour.

2. No, this is only one transaction. For Service Bus you get charged a transaction each time you put a message on a queue and each time a message is requested (which can be a little opaque given that Recieve makes calls multiple times in the background). Note that the docs point out that you get charged for each idle transaction (meaning a null result from a Receive call).

3. Again, you can implement a back off methodology so that you aren't hitting the queue so often. Another suggestion I've recently heard was if you have a queue that isn't seeing a lot of traffic you could also check the queue depth to see if it was > 0 before entering the loop for processing and if you get no messages back from a receive call you could go back to watching the queue depth. I've not tried that and it is possible that you could get throttled if you did the queue depth check too often I'd think.

If these are your production numbers then your subscription isn't really processing a lot of messages. It would likely be a really good idea to have a back off policy to a time that is acceptable to wait before it is processed. Like, if it is okay that a message sits for more than 10 minutes then create a back off approach that will eventually just be checking for a message every 10 minutes, then when it gets one process it and immediately check again.

Oh, there is a Receive overload that takes a timeout, but I'm not 100% that is a server timeout or a local timeout. If it is local then it could still be making the calls every X seconds to the service. I think this is based on the OperationTimeout value set on the Messaging Factory Settings when creating the SubscriptionClient. You'd have to test that.