How to resolve jms server performance issue when client uses temporary replyTo queue?

Question

I am currently building a Mule ESB server application which uses a request-response jms connector. Since it is being used in a highly concurrent environment, we enabled spring jms cache in our MQ config.

<spring:beans>
    <mule>
        <!-- MQ Factory -->
        <spring:bean id="testMsgMqFactoryBean1" name="testMsgMqFactory1" class="com.ibm.mq.jms.MQQueueConnectionFactory">
            <spring:property name="channel" value="${test.msg.mq.channel.1}" />
            <spring:property name="queueManager" value="${test.msg.mq.queueManager.1}" />
            <spring:property name="hostName" value="${test.msg.mq.hostName.1}" />
            <spring:property name="port" value="${test.msg.mq.port.1}" />
            <spring:property name="transportType" value="${mq.jms.transportType}" />
        </spring:bean>
        <spring:bean id="testMsgMqFactoryBeanCache1" class="org.springframework.jms.connection.CachingConnectionFactory">
            <spring:property name="targetConnectionFactory" ref="testMsgMqFactoryBean1" />
            <spring:property name="sessionCacheSize" value="${test.threading.profile.maxThreadsActive}" />
            <spring:property name="cacheConsumers" value="false" />
            <!-- <spring:property name="cacheProducers" value="false" /> -->
        </spring:bean>
        <!-- MQ Connector 1 -->
        <jms:custom-connector name="testMsgMqConnector.1" class="org.mule.transport.jms.websphere.WebsphereJmsConnector" doc:name="Custom JMS">
            <spring:property name="specification" value="1.1" />
            <spring:property name="connectionFactory" ref="testMsgMqFactoryBeanCache1" />
            <spring:property name="persistentDelivery" value="false" />
            <spring:property name="disableTemporaryReplyToDestinations" value="true" />
            <spring:property name="numberOfConsumers" value="${test.threading.profile.maxThreadsActive}" />
            <spring:property name="maxRedelivery" value="-1" />
            <receiver-threading-profile maxThreadsActive="${test.threading.profile.maxThreadsActive}" maxBufferSize="${test.threading.profile.maxBufferSize}" maxThreadsIdle="${test.threading.profile.maxThreadsIdle}"/>
            <reconnect frequency="${mq.jms.reconnection.frequency}" count="${mq.jms.reconnection.count}" blocking="false" />
        </jms:custom-connector>

        <!-- msgworks inbound and outbound MQ setup -->
        <!-- Rewards -->
        <jms:endpoint exchange-pattern="request-response" queue="${test.msg.mq.inbound.account.queue}" name="testQueue1" connector-ref="testMsgMqConnector.1" doc:name="JMS" />
    </mule>
</spring:beans>

This configuration runs fine when the client uses a static replyTo queue. However, we have some customers who are using dynamic/temporary replyTo queue. Since org.springframework.jms.connection.CachingConnectionFactory caches producers, for every temporary replyTo queue, a producer object is cached and never closed. After processing hundreds of requests, the application started throwing exceptions:

********************************************************************************
Message               : Failed to create and dispatch response event over Jms destination "queue://QMGR1/TESTret5a975v53AF980F2006BE02?targetClient=1". Failed to route event via endpoint: null. Message payload is of type: JMSTextMessage
Code                  : MULE_ERROR-42999
--------------------------------------------------------------------------------
Exception stack is:
1. MQJE001: Completion Code 2, Reason 2017 (com.ibm.mq.MQException)
  com.ibm.mq.MQQueueManager:2808 (null)
2. MQJMS2008: failed to open MQ queue TESTret5a975v53AF980F2006BE02(JMS Code: MQJMS2008) (javax.jms.ResourceAllocationException)
  com.ibm.mq.jms.MQQueueServices:398 (http://java.sun.com/j2ee/sdk_1.3/techdocs/api/javax/jms/ResourceAllocationException.html)
3. Failed to create and dispatch response event over Jms destination "queue://QMGR1/TESTret5a975v53AF980F2006BE02?targetClient=1". Failed to route event via endpoint: null. Message payload is of type: JMSTextMessage (org.mule.api.transport.DispatchException)
  org.mule.transport.jms.JmsReplyToHandler:173 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/api/transport/DispatchException.html)

After investigating the MQ error code (MQJE001: Completion Code 2, Reason 2017), I found that the reason behind this error is because we never closed producers, and the producers exhausted MQ Handles on the Queue Manager. The quick and easy fix is to uncomment the line in spring jms cache config to close Producers each and every time.

<spring:bean id="testMsgMqFactoryBeanCache1" class="org.springframework.jms.connection.CachingConnectionFactory">
    <spring:property name="targetConnectionFactory" ref="testMsgMqFactoryBean1" />
    <spring:property name="sessionCacheSize" value="${test.threading.profile.maxThreadsActive}" />
    <spring:property name="cacheConsumers" value="false" />
    <spring:property name="cacheProducers" value="false" />
</spring:bean>

Now I am not seeing the MQ issue, but came up with another performance issue, because no producers are cached, so a new producer is created every single time.

My question is, how to deal with this scenario? Since the client won't change the way they receive reply messages from temporary queues, how can we avoid exhausting MQ handlers without impacting the performance.

Thank you very much - Lei

Answer 1

This is a very interesting use case. However, I'm afraid there is nothing out of the box to fix this. There are the more obvious solutions: disable the caching or extend the spring cache provider.

Temporary queues and performance are definitely not two things that you can have at the same time. I would suggest another posibility:

If you are using the temporary queues to have the responses come back to a given consumer only, probably in adition to having old messages discarded on reconnection:

You could use a well know queue for the replies, using a combination of a header with the hostname that should receive the message plus a selector different on each consumer node and a TTL on the messages sent to make them dissapear after a while.