How should I handle "java.net.SocketException: Connection reset" in multithread AWS S3 file upload?

Question

I have a ThreadPoolExecutorService to which I'm submitting runnable jobs that are uploading large (1-2 GB) files to Amazon's S3 file system, using the AWS Java SDK. Occasionally one of my worker threads will report a java.net.SocketException with "Connection reset" as the cause and then die.

AWS doesn't use checked exceptions so I actually can't catch SocketException directly---it must be wrapped somehow. My question is how I should deal with this problem so I can retry any problematic uploads and increase the reliability of my program.

Would the Multipart Upload API be more reliable?

Is there some exception I can reliably catch to enable retries?

Here's the stack trace. The com.example.* code is mine. Basically what the DataProcessorAWS call does is call putObject(String bucketName, String key, File file) on an instance of AmazonS3Client that's shared across threads.

14/12/11 18:43:17 INFO http.AmazonHttpClient: Unable to execute HTTP request: Connection reset
java.net.SocketException: Connection reset
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
    at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:377)
    at sun.security.ssl.OutputRecord.write(OutputRecord.java:363)
    at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:830)
    at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:801)
    at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)
    at org.apache.http.impl.io.AbstractSessionOutputBuffer.write(AbstractSessionOutputBuffer.java:169)
    at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:119)
    at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:102)
    at com.amazonaws.http.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:153)
    at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:98)
    at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108)
    at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:122)
    at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:271)
    at org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestEntity(ManagedClientConnectionImpl.java:197)
    at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:257)
    at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doSendRequest(SdkHttpRequestExecutor.java:47)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
    at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:685)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:460)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:295)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3697)
    at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1434)
    at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1294)
    at com.example.DataProcessorAWS$HitWriter.close(DataProcessorAWS.java:156)
    at com.example.DataProcessorAWS$Processor.run(DataProcessorAWS.java:264)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Answer 1

For that you have to use only AmazonS3Client not Trasfermanager for upload and download

U have to configure AmazonS3Client with following properties:

1. connectionTimeout=50000 in ms
2.maxConnections=500
3.socketTimeout=50000 in ms
4.maxErrorRetry=10

For upload use

AmazonS3Client.putObject(bucketName, key, inputFile);

For download use

S3Object s3Object = AmazonS3Client.getObject(new GetObjectRequest(bucketName, key));`
InputStream downloadStream  = s3Object.getObjectContent();  

and store that stream by reading bytes.

Answer 2

'Connection reset' means the connection is closed. Close the socket, or whatever higher-level construct you're using. Probably the server has decided the upload is too large, or it's overloaded, or something. Whether you can retry the operation is something only you can know.