How can I improve performance for an MQ based batch application?

Question

I have an application where messages keep coming at a rate of 70K XMLs per hour. We consume these XML messages and store it into an intermediate queue. The intermediate queue is created because we need to meet SLA of consuming all the messages with 24 hours. We are able to consume and load the XMLS into the internal queue within 24 hours. After loading it to internal queue, we process the XMLS (parse, apply very few transformation, perform very few validations) and store the data to a heavily normalized data model. I know that the datamodel can have a huge impact on performance, unfortunately, we have no control over the datamodel. Currently, we take 3.5 minutes to process 2K messages, which is unacceptable. We want to bring it down to 1 minute for 2K messages. Here is what we have done so far:

1) Applied indexes wherever applicable.
2) Use XMLBeans for parsing the XMLs (size of each XML is not very huge)
3) Removed all unnecessary validations, transformatios, etc.

The application runs on:
Operating system: RHEL 5.4 64 bit
Platform: JDK 1.6.0_17, 64 bit
Database: Oracle 11g R2 64 bit (2 node cluster)
External MQ: IBM Queue
Internal temporary storage MQ: JBoss MQ
Application Server: Jboss 5.1.0.GA (EAP Version)

The order in which we consume and process the XML messages is very important and so we cannot do a parallel processing.

Is there anything else we can do to improve performance?

Answer 1

Some suggestions outside of message delivery tuning since it appears this is not your [primary] bottleneck:

• You mentioned you are storing data into a highly normalized database. This invariably means one or more reference data or PK lookups which creates several additional trips to the database to fetch this data. To avoid or reduce this, create a local cache with all your reference data and update the cache as you go. In memory lookups will be significantly faster than a trip to the DB.
• If you feel you have insufficient RAM to cache all your decodes and reference data, shoot for a disk based cache (e.g. EHCache which will do RAM, Disk or Overflow) or a lightweight local DB like HyperSonic or H2 which will still give you better lookup times than a trip to Oracle (unless you're on the same host, and even then....)
• Ultimately, if each message requires many round trips to the DB, you may benefit from migrating the processing of the message to the DB itself, where you can implement the process in PL/SQL or Java.
• If your write to the database for one message processed involves multiple inserts/updates, make sure to use prepared statement batching. This will send multiple inserts/updates in one call to the DB.
• Speaking of prepared statements, make sure your JBoss DataSource configuration for Oracle has prepared-statement-cache-size set to some number high enough to handle all your prepared statements created during processing (and not the default which is zero, or no caching).
• The XML parser you are using may be imposing more overhead than is necessary, even (or especially) for small messages. If you are using JAXB, make sure you're not recreating the unmarshaller more than once (or more than necessary). Alternatively, try a Pull/Streaming parser. If you are using a DOM parser, the additional memory required may be causing a lot of garbage collection.
• Silly thing, but worth mentioning, if you are executing a lot of logging for each message, that will be costing you time, so turn it off.
• Using JBoss MQ as an intermediary buffer is elegant but it is probably not the fastest way to store your messages for deferred processing since the persistence is more complex and generalized for all sorts of JMS message types. On that note, if JBoss MQ is persisting to Oracle anyways, then it seems improbable that a custom persistence procedure would not be faster. If JBoss MQ is storing to HyperSonic (as it does by default), you can still probably outperform the store of the JMS message with some custom code. This will also mean that you will need a new mechanism to pull the message back out of the DB for processing, but as with the JMS store, a custom process may outperform the more generalized procedure implemented by JBoss MQ.
• Storing intermediary messages to the DB may also give more query flexibility to determine where messages do not have to be serially processed. (Perhaps, for example, messages originating from different clients do not need to be processed in sequence). Of course, you can also do this with JBoss MQ by placing the appropriate headers in the intermediary messages. This would allow you to parallelize by using different selectors in multiple different message listeners/processors.

One quick item on messaging.....

You did not mention if you were using message driven beans with WebSphere MQ, but if you are, there is a setting in the Inbound Configuration called pollingInterval which, to quote from the docs, means:

If each message listener within a session has no suitable message on its queue, this is the maximum interval, in milliseconds, that elapses before each message listener tries again to get a message from its queue. If it frequently happens that no suitable message is available for any of the message listeners in a session, consider increasing the value of this property. This property is relevant only if TRANSPORT has the value BIND or CLIENT.

The default pollingTime is 5000 ms. Your current message processing time is

(3.5 * 60 * 1000 / 2000)

= 105 ms per message.

If you introduce a 5000 ms pause here-and-there, that will seriously cut down on your throughput, so you might want to look into this by measuring the ongoing difference between the message enqueue time and the time that you receive the message in your JBoss message listener. The enqueue time can be determined from these message headers:

• JMS_IBM_PutDate
• JMS_IBM_PutTime

All in all, your best bet is going to be to figure out how to parallelize.

Good luck.

//Nicholas

Answer 2

WebSphere MQ, even on a small server, can unload messages MUCH faster than the rate you describe. The Windows Performance Report for WMQ V7 tested at more than 2,200 2k persistent round trips (one request and one reply) per second over client channels. That's more than 4,000 messages per second.

The bottleneck in your case would seem to be the latency of processing messages and the dependency on processing the messages in a particular order. The option that could give you the MOST performance boost would be to eliminate the order dependency. When I worked at a bank we had a system that posted transactions in the exact order they arrived and everyone said this requirement was mandatory. However, we eventually revised the system to perform a memo-post during the day and then repost in a later step. The memo-posting occurred in any order and supported parallelism, failover and all the other benefits of multi-instance processing. The final post applied the transactions in logical order (and in fact in an order that was most favorable to the customer) once they were all in the DB. Sequence dependencies lock you into a singleton model and are a worst-case requirement for asynch messaging. Eliminate them if at all possible.

The other area for improvement will be in the parsing and processing of the messages. As long as you are stuck with sequence dependencies, this is the best bet for improving performance.

Finally, you always have the option to throw money at the problem in the form of more memory, CPU, faster disk I/O and so forth. Essentially this is addressing software architecture with horsepower and is never the best solution but often it buys you enough time to address the root cause.