Android AWS s3 SDK download throws "Socket is closed" exception or terminates early

Question

I'm trying to download an object from S3 using the AWS Android SDK 1.0.4 or 1.0.3.

This is my code:

AmazonS3Client client = getConnection(userCredentials);
S3Object obj = client.getObject(workspaceName, objectName);
ObjectMetadata meta = obj.getObjectMetadata();
long size = meta.getContentLength();
logger.info("S3 object length: "+size);
InputStream is = new BufferedInputStream(obj.getObjectContent());
byte[] fragmentBytes = IOUtils.toByteArray(is);
logger.info("Read bytes: "+ fragmentBytes.length);

This sometimes, rarely, works. Most of the time either an "java.net.SocketException: Socket is closed" is thrown or no error is reported but only part of the object is read.

Note, that the BufferedInputStream should not be necessary for IOUtils.toByteArray(...) and it makes no difference if it is there or not.

The problem does not seem to occur when stepping through the code line by line in a debugger.

This happens on my Android 2.3.3 device and 3.2 device. It happens over WiFi and 3G.

Any ideas?

ps> The objects are about 250k big

Answer 1

The issue is that the client is being Garbage Collected. To fix it, you need to keep an explicit reference to the S3Client.

The flow of the bug is as follows:

1. The program gets an S3 client
2. The program gets a S3 object from the client.
3. The program gets an inputStream from the s3Object, which gets it through the client.
4. Garbage collection runs, and since there is no reference to the client, it gets Garbage Collected, randomly closing your inputStream.

IMO, this is a very bad bug by Amazon. Why the S3 object input stream does not have a reference back to the client is a mystery to me.

Here's the thread on the AWS forums: https://forums.aws.amazon.com/thread.jspa?threadID=83326

Answer 2

You probably already solved this but I could not find the correct answer and ended up figuring it out by myself, so here goes the solution:

The getObjectContent methods documentation is very clear about closing the InputStream:

S3ObjectInputStream com.amazonaws.services.s3.model.S3Object.getObjectContent()

Gets an input stream containing the contents of this object. Callers should close this input stream as soon as possible, because the object contents aren't buffered in memory and stream directly from Amazon S3.

You are making the same mistake as I was of using the InputStream (S3Object) as if closure was transparent.

I am guessing you need to get an image from S3 to be used on Android, so here goes an example method that returns a Bitmap. It really applies to any kind of input.

private Bitmap getFromStore(String fileName) {
    AmazonS3Client client = new AmazonS3Client(new BasicAWSCredentials(ACCESS_KEY, SECRET_KEY));

    S3Object o = client.getObject(BUCKET, fileName);
    InputStream is = o.getObjectContent();

    Bitmap image = null;

    try {   
        // Use the inputStream and close it after.
        image = BitmapFactory.decodeStream(is);
    } finally {
        try {
            is.close();
        } catch (IOException e) {
            Log.w(TAG, "Error trying to close image stream. ", e);
        }
    }

    return image;
}

Just in case, dont forget to execute this on a different thread then the UI.

Good luck,