Apache Camel: Aws-S3 consumer starts failing with connection pool timeout

Question

I am using AWS-S3 consumer to poll files on a certain location on S3 at regular intervals. After polling for certain no of times, it starts failing with exceptions as given,

Will try again at next poll. Caused by:[com.amazonaws.AmazonClientException - Unable to execute HTTP request:
Timeout waiting for connection from pool]
com.amazonaws.AmazonClientException: Unable to execute HTTP request:Timeout waiting for connection from pool
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:376) ~[aws-java-sdk-1.5.5.jar:na]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:202) ~[aws-java-sdk-1.5.5.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3037) ~[aws-java-sdk-1.5.5.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3008) ~[aws-java-sdk-1.5.5.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:531) ~[aws-java-sdk-1.5.5.jar:na]
at org.apache.camel.component.aws.s3.S3Consumer.poll(S3Consumer.java:69) ~[camel-aws-2.12.0.jar:2.12.0]
at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187) [camel-core-2.12.0.jar:2.12.0]
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114) [camel-core-2.12.0.jar:2.12.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_60]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_60]

From what I understand, the reason shall be the consumer exhausting the available connections from the pool as it uses a new connection every poll. What I need to know is how to release the resources after every poll and why does the component itself doesn't do it.

Camel Version: 2.12

Edit: I modified the consumer to pick custom S3 client with specific connection timeout, maxconnections, maxerrorretry and sockettimeout, but of no use. Resultant is same.

S3 Client configuration:

ClientConfiguration clientConfiguration = new ClientConfiguration();
    clientConfiguration.setMaxConnections(50);
    clientConfiguration.setConnectionTimeout(6000);
    clientConfiguration.setMaxErrorRetry(3);
    clientConfiguration.setSocketTimeout(30000);
    main.bind("s3Client", new AmazonS3Client(awsCredentials, clientConfiguration));

The object of AmazonS3Client named "s3Client" is bounded to the Camel context and is provided to Camel AWS-S3 component based route. Now, Camel on its own manages this resource.

Required solution: Am expecting solution specific to Camel Aws-S3 consumer and not generic Java solution as am aware that connection shall be closed after its task is done for it to be released and used again. What am confused about is why is Camel not doing this automatically when provided with the connection pool or if I am missing any configuration specifically.

Answer 1

Camel Consumer class opens connection for each "Key" and creates an exchange out of it. This exchange is forwarded on to the route for processing but never closed automatically, even on calling "Stop". Resultant, the connection pool runs out of free connections. What needs to be done is extract the S3ObjectInputStream out of the exchange and close it.

S3ObjectInputStream s3InputStream = exchange.getIn().getBody(S3ObjectInputStream.class);
s3InputStream.close();

The answer is pretty much close to what the others suggest that is to close the connection. But as explained, Camel specific answer was expected and an explanation to why doesn't Camel handles this on its own.

Answer 2

Any where Connection Pooling concept is the same.If you not able to get a connection even it is idle,in development purpose we need to explicitly call close() by checking whether connection Idle. for exmaple:

 if(con.isIdle()&& !con.closed()){
con.close();
}

Then only we'll get the conncetions available.Even though most frameworks do it its better to finalise this code from our connectionFactory classes.

Edit:

https://forums.aws.amazon.com/message.jspa?messageID=296676

This link will surley helps you in getting your specific answer as you didn't specify your code of S3Object connection class.

Edit 2 :

Try this method in your ClientConfiguration

public ClientConfiguration withConnectionMaxIdleMillis(long connectionMaxIdleMillis)

this might resolve your error because it closes the connection automatically if there is Idle connection the pool and not reused.