Performance degrade issue using ActiveMQ scheduler

Question

In one of our project we are suffering for ActiveMQ performance degrade when we use the schedule facility (enabling schedulerSupport in ActiveMQ configuration) with high amount of message (1M or more). The project is Spring based, and we are, also, using Camel for routes management.

Our routes are configured as follow:

• a producer P produces 800 messages/sec and sends them through a non persistent ActiveMQ queue Q
• a consumer Cq processes messages from Q according to the following rule: for each message it verifies if the message can be process immediately (every message has a time-window attribute to define that). If it's possible, Cq forwards the message to the Throttling queue T; if it's not, Cq forwards the messages into a support queue S
• a consumer Cs processes messages from S and it schedules the message according to its time-window (setting the ActiveMQ property ScheduledMessage.AMQ_SCHEDULED_DELAY with the following statement: message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, <delay>);) where <delay> is evaluated according to the message time-window
• when a scheduled message wakes up from the ActiveMQ scheduled-area, it is forwarded to the Throttling queue T
• the consumer Ct processes the message from T using a third parties component (via TCP connection)

When P produces messages for immediate processing (i.e., without time-window) the system performs well even with an high amount of messages (1M or more): monitoring the queues we can see them almost empty (no message stays pending in any queue).

When P produces messages for delayed processing (i.e., with time-window) we see the following behavior:

• we have messages pending in all the queues (Q and S), while we expect to have them pending only on the queue S
• when the message scheduled in S wakes up they moves to the T queue: these messages are processed according to the throttling configuration (1K messages/sec)
• the message still pending in Q and S move very slowly to T (around 50 messages/sec)
• in case we try a second round of messages (without time-window) just after all the precious messages have been processed (using, again the producer P), we still sending out messages to 50 messages/sec even if, in this case, we are not using the scheduled-area at all

Summary of the issue

• ActiveMQ version: 5.9.0 (configured with 3GB of max memory limit)
• Camel version: 2.15.1
• Spring version: 4.1.3.RELEASE
• average dimension of 1 message: 4.5KiB
• Critical routes:
  • Q -> S (scheduled-area) -> T
  • Q -> T (after the use of the scheduled-area)
• Without using scheduler: within 1M messages, we process easily 1K messages/sec (the limit depends from a Camel Throttling queue)
• Using scheduler: within 1M messages, we process 50 messages/sec
• Monitoring CPUs and memory usage, we cannot spot-out any issue: from 2% to 5% of CPUs usage and we never go out-of-memory (the Java GC seems works fine too)
• Hardware (virtualized based on VMWare):
  • CPUs: 24 vCPUs mapped on 24 real 'Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz' CPUs
  • Ram: 31744MB

cheers and thanks in advance for your help,

Eros

Answer 1

ActiveMQ from version 5.4 has an optional persistent scheduler built into the ActiveMQ message broker activemq.apache.org/delay-and-schedule-message-delivery.html

So to improve performance try to add this code below to the brokerURL on your ActiveMQConnectionFactory

For example :

ConnectionFactory connectionFactory = new ActiveMQConnectionFactory("tcp://localhost:61616?jms.copyMessageOnSend=false&jms.dispatchAsync=true&jms.useAsyncSend=true");

take a look at

http://activemq.apache.org/connection-configuration-uri.html http://activemq.apache.org/performance-tuning.html