how to create multiple instances of activemq topic subscribers using virtual destinations?

Question

I have a publisher that is pushing messages to a topic. I have multiple subscribers each doing a different task once they consume the message from the topic. Now I want my system to scale to multiple instances of the same process running on different hosts/same host. e.g. I want to run multiple copies of my application A on different hosts so that if one instance of A is slow, then the other instances can pull in subsequent messages and make forward progress.. I found out that this is possible using virtual destinations. I followed the steps here - http://activemq.apache.org/virtual-destinations.html

But how do i setup my multiple subscribers to the same topic with the same client id? when i try to do that, i get errors. when i try some other way, it doesn't work. can someone help?

Normally, I start a subscriber by doing the below steps -

        ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(ActiveMQConnection.DEFAULT_USER, ActiveMQConnection.DEFAULT_PASSWORD, ActiveMQConnection.DEFAULT_BROKER_URL;);
        activeMQConnection = connectionFactory.createConnection();

        activeMQConnection.setClientID("subscriber1");
        activeMQConnection.setExceptionListener(exceptionListener);
        activeMQSession = activeMQConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE);
        activeMQTopic = activeMQSession.createTopic("myTopic");            
        activeConsumer = activeMQSession.createDurableSubscriber(activeMQTopic, "myTopic");            
        activeConsumer.setMessageListener(messageListener);
        activeMQConnection.start();

when i try to create a 2nd subscriber and pass the topic name as "VirtualTopic.myTopic", nothing happens.

thanks

Answer 1

The virtual topics feature is very simple and quite powerful once you understand it.

1. When using virtual topics - there is no need for durable consumers. That is because for each client you will get an instance of regular queue created. If you have 5 clients (application A, B, C, D, E) you will get 5 queues created and populated with the copy of the messages every time message is sent to the virtual topic.

2. Actually it is a limitation of durable consumer - that only ONE connection is allowed per clientId. Being a regular queue, you can create as many consumers as you like and queue will guarantee that 1 message will be received only by 1 consumer. So if you have application A that takes 1 minute to process a message, you can create 5 instances of it listening to the same queue. When you will post 5 messages within 1 second, each of your application will receive its own message to process.

3. There are not well documented requirements which are not intuitive. To make virtual topic work you need

   • Use VirtualTopic. in your topic name, for example VirtualTopic.Orders (this prefix can be configured)
   • Use Consumer. in the name of the queue you. Like Consumer.ApplicationA.VirtualTopic.Orders where ApplicationA is actually your client id
   • Use regular subscribers not durable ones for the queue above.

Example:

string activeMqConsumerTopic = "Consumer.AmqTestConsumer.VirtualTopic.Orders";

IQueue queue = SessionUtil.GetQueue(session, activeMqConsumerTopic);

IMessageConsumer consumer = session.CreateConsumer(queue);

Queue is created for automatically whenever the first instance of consumer is subscribed to it. Since that moment all messages that are sent to topic are duplicated/copied into all associated queues.

Hope this helps.

Answer 2

Virtual Topics is the answer for you. However you have to define a naming standard for all virtual topic queues. Here is the answer for this:

Virtual Topics helps with following prospective: 1. Load Balancing of messages 2. Fast Failover of Subscriber 3. Re-using same connection Factory for different Producers and Consumers. (Durable Subscribers needs a Unique JMS Client Id and same cannot be reused for any other Producer or consumer)

here is the way to do it, below example creates prefix VTCON.*. So every queue with this prefix and Topic Name at the end will consumer the message.

<virtualDestinations> <virtualTopic name="TEST.TP01" prefix="VTCON.*." selectorAware="false"/> </virtualDestinations>

http://workingwithqueues.blogspot.com/2012/05/activemq-virtual-topics-or-virtual.html