AWS SDK S3 Socket Closed exception

Question

My application uses close to 10 threads, each of which makes perhaps 7,000 Put Requests to S3 per minute. (I'm running it on a powerful EC2 box which can handle the load quite well.) It runs beautifully for close to an hour, but, after an hour, gets Unable to execute HTTP request: Socket Closed exceptions:

        http.AmazonHttpClient: Unable to execute HTTP request: Socket Closed
    java.net.SocketException: Socket Closed
    at java.net.AbstractPlainSocketImpl.setOption(AbstractPlainSocketImpl.java:206)
    at java.net.Socket.setSoTimeout(Socket.java:1105)
    at sun.security.ssl.SSLSocketImpl.setSoTimeout(SSLSocketImpl.java:2414)
    at org.apache.http.impl.io.SocketInputBuffer.isDataAvailable(SocketInputBuffer.java:106)
    at org.apache.http.impl.AbstractHttpClientConnection.isResponseAvailable(AbstractHttpClientConnection.java:246)
    at org.apache.http.impl.conn.ManagedClientConnectionImpl.isResponseAvailable(ManagedClientConnectionImpl.java:180)
    at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
    at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doSendRequest(SdkHttpRequestExecutor.java:47)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
    at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:713)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:518)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:446)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:256)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3641)
    at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1438)
    at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInOneChunk(UploadCallable.java:128)
    at com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:120)
    at com.amazonaws.services.s3.transfer.internal.UploadMonitor.upload(UploadMonitor.java:176)
    at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:134)
    at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:50)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

The Put Requests are done asynchronously, using the AWS SDK TransferManager. I imagine that, in the time it takes for one put request to fully complete, about 10 have been made asynchronously.

Googling that exception, I found two possible causes:

1. The limit on MaxConnections. I've raised it from the default 50 to 3000, to no avail.
2. Premature garbage collecting. I've tried keeping a reference to the Upload objects returned by TransferManager (in an concurrent queue), and, again, no help.

How can I fix this? Again, the app runs well for close to an hour, but, consistently, hits this wall after about an hour. (I'm running on Amazon AMI Linux on EC2.)

Update

• No code other than the AWS SDK touches the sockets, or even knows about them. All the HTTP work is done exclusively through AWS SDK.
• So, if something's closing them, it must be something in the AWS SDK.
• The code is running on an EC2 server; there's no reason to expect any type of network connectivity issues between EC2 and S3, and certainly no reason they should happen predictably (after an hour into the run) each time

Answer 1

I'm not sure if this is the answer, but http://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html states that "if you expect a rapid increase in the request rate for a bucket to more than 300 PUT/LIST/DELETE requests per second or more than 800 GET requests per second, we recommend that you open a support case to prepare for the workload and avoid any temporary limits on your request rate". Perhaps since I exceeded the limit, AWS starts aborting connections; the SDK, detecting the IDLE sockets, closes them, and, voila!, we get exceptions.

UPDATE: Not sure if this is correct. Amazon seems to state that, in this case, you'll get an explicit "Slow Down" error message, not an unexpected close. So, the puzzle remains.

Answer 2

The Exception is SocketException caused by setSoTimeout() method in java.net.Socket.(See the stack trace). The method can be viewed here: http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/net/Socket.java#Socket.setSoTimeout%28int%29

Possible reason may be that the requests to S3 are still pending/incomplete resulting the thread to wait(). Once the wait time exceeds socket timeout , the socket is closed and exception is thrown.