How to decide optimal settings for setMaxTotal and setDefaultMaxPerRoute?

Question

I have a RestService running on 45 different machines in three datacenters (15 in each datacenter). I have a client library which uses RestTemplate to call these machines depending on where the call is coming from. If the call is coming from DC1, then my library will call my rest service running in DC1 and similarly for others.

My client library is running on different machines (not on same 45 machines) in three datacenters.

I am using RestTemplate with HttpComponentsClientHttpRequestFactory as shown below:

public class DataProcess {

    private RestTemplate restTemplate = new RestTemplate();
    private ExecutorService service = Executors.newFixedThreadPool(15);

    // singleton class so only one instance
    public DataProcess() {
        restTemplate.setRequestFactory(clientHttpRequestFactory());
    }

    public DataResponse getData(DataKey key) {
        // do some stuff here which will internally call our RestService
        // by using DataKey object and using RestTemplate which I am making below
    }   

    private ClientHttpRequestFactory clientHttpRequestFactory() {
        HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory();
        RequestConfig requestConfig = RequestConfig.custom().setConnectionRequestTimeout(1000).setConnectTimeout(1000)
                .setSocketTimeout(1000).setStaleConnectionCheckEnabled(false).build();
        SocketConfig socketConfig = SocketConfig.custom().setSoKeepAlive(true).setTcpNoDelay(true).build();

        PoolingHttpClientConnectionManager poolingHttpClientConnectionManager = new PoolingHttpClientConnectionManager();
        poolingHttpClientConnectionManager.setMaxTotal(800);
        poolingHttpClientConnectionManager.setDefaultMaxPerRoute(700);

        CloseableHttpClient httpClientBuilder = HttpClientBuilder.create()
                .setConnectionManager(poolingHttpClientConnectionManager).setDefaultRequestConfig(requestConfig)
                .setDefaultSocketConfig(socketConfig).build();

        requestFactory.setHttpClient(httpClientBuilder);
        return requestFactory;
    }

}

And this is the way people will call our library by passing dataKey object:

DataResponse response = DataClientFactory.getInstance().getData(dataKey);

Now my question is:

How to decide what should I choose for setMaxTotal and setDefaultMaxPerRoute in PoolingHttpClientConnectionManager object? As of now I am going with 800 for setMaxTotal and 700 setDefaultMaxPerRoute? Is this a reasonable number or should I go with something else?

My client library will be used under very heavy load in multithreading project.

Answer 1

There are no formula or a recipe that one can apply to all scenarios. Generally with blocking i/o one should have approximately the same max per route setting as the number of worker threads contending for connections.

So, having 15 worker threads and 700 connection limit makes little sense to me.

Answer 2

Apparently there is not a definite and worthy formula into the situation. The relation between poolsize and throughput and response-time is not simple enough. One particular case which comes to mind is that after a certain value the response-time starts increasing on increasing the pool-size.

Generally with blocking i/o one should have approximately the same max per route setting as the number of worker threads contending for connections.