Apache Camel Route performance degradation under load

Question

I'm using Apache Camel to route a SOAP request based on a certain attribute in the request message. The message is matched against a regex and if a match is found the request will be routed to "calldestination1" and if not, it will be routed to "calldestination2".

I'm using the following configuration:

<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:camel="http://camel.apache.org/schema/spring"
    xmlns:cxf="http://camel.apache.org/schema/cxf"
    xmlns:context="http://www.springframework.org/schema/context"
    xsi:schemaLocation="
    http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
    http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
    http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd
    http://camel.apache.org/schema/cxf http://camel.apache.org/schema/cxf/camel-cxf.xsd">

<!-- ... -->

<cxf:cxfEndpoint id="testEndpointTest"
    address="http://localhost:8080/testEndpoint"
    endpointName="s:testEndpoint_Port"
    serviceName="s:testEndpoint"
    wsdlURL="wsdl/testEndpoint.wsdl"
    xmlns:s="http://teste.com/testEndpoint"/>

<!-- ... -->

<camelContext xmlns="http://camel.apache.org/schema/spring">

    <endpoint id="calldestination1" uri="http://localhost:8080/destination1?bridgeEndpoint=true&amp;throwExceptionOnFailure=false"/>
    <endpoint id="calldestination2" uri="http://localhost:8080/destination2?bridgeEndpoint=true&amp;throwExceptionOnFailure=false"/>

    <route streamCache="true">
        <!--CXF consumer using MESSAGE format--> 
        <from uri="cxf:bean:testEndpointTest?dataFormat=MESSAGE"/>

        <choice>
            <when>
                <simple>${bodyAs(java.lang.String)} regex ${properties:router.regex}</simple>
                <to uri="calldestination1"/>
            </when>
            <otherwise>
                <to uri="calldestination2"/>
            </otherwise>
        </choice>

    </route>

</camelContext>

When the destination server, where "calldestination2" runs, is under load the requests can take around 1150ms to respond. Apache Camel does not seem to handle this very well.

To replicate this behavior I used SoapUI with a SOAP MockService with a delay (OnRequest Script) and jmeter. Fist I ran the test against SoapUI MockService with no delay and then with a 1100ms delay.
Then I configured Apache Camel to route the request to SoapUI service and repeated the tests.

JMeter -> SoapUI - 0ms delay
~1200 requests per second; 25ms request average; 0% Errors

JMeter -> SoapUI - 1100ms delay
~100 requests per second; 1128ms request average; 0% Errors

JMeter -> Apache Camel -> SoapUI - 0ms delay
~420 requests per second; 285ms request average; 0% Errors

JMeter -> Apache Camel -> SoapUI - 1100ms delay
~8 requests per second; 14800ms request average; 97.23% Errors by timeout

The timeout in Apache Camel is set to 30 seconds.

Why is Apache Camel having such a low performance in the last case and how can I improve the it?

EDIT 1:
I created a repository in GitHub that contains the Apache Camel project, SoapUI mock service and jmeter tests for easy testing.

https://github.com/jraimundo/apache-camel-route-tester

Answer 1

Basic problem

Such problems are always a problem of resources. As long as all components have enough resources and answer fast, all is fine. As soon as one of them encounters a resource limitation it becomes slow.

In the JMeter-SoapUI scenario, the intentional latency of SoapUI is handled by JMeter. Because SoapUI takes more than a second to respond, the JMeter requests are held open for this time. If the JMeter thread pool for requests is exhausted (all threads are waiting for an answer from SoapUI), it cannot further scale. Based on your measures, the thread pool size could be 100.

Then you put Camel in the middle. With this you introduce new thread pools. There must be one to receive requests (CXF) and probably one that sends requests (Camel HTTP). Now the SoapUI latency must also be handled by these pools. Same situation, but now the thread pools of the Camel component are the limitation.

Let's assume the thread pool for Camel HTTP requests is 10 by default. JMeter begins to send requests. If JMeter sends new requests faster than SoapUI responds, the 10 threads to send HTTP requests to SoapUI are very fast all busy (waiting for SoapUI).

New requests of JMeter arrive, but no new HTTP requests to SoapUI are possible until until one of the threads is free again. Around 8 parallel requests (from your measures) seems to be reasonable in this case.

So it is obvious that if you want to serve 100 requests per second in a scenario like this you need to tune all involved thread pools to handle this. And you must also fine tune the different timeouts (CXF, Camel HTTP).

Your code

One point I noticed in your code is that you use the Camel HTTP component for your target endpoints. That component uses the Apache HTTP client 3.x.

If you want to use a more current Apache HTTP client, you must use the Camel HTTP4 component (4 because it uses Apache HTTP client 4.x). I don't know if it makes a big difference, but the old version is declared as "end of life" since years.

Another thing are the timeouts. You write that you set the Camel timeout to 30 seconds. But that is probably not the timeout of CXF or the Apache HTTP client. HTTP clients have multiple timeouts: it can take too long to establish a connection and it can take too long to receive a response.