iOS Amazon S3 download large files

Question

I'm quite new to Amazon S3 and I'm having difficulty downloading large files from S3.

I have successfully downloaded a file that is 35MB every time, but when the size of the file is really big around 500 MB - 1.7GB the application crashes.

When trying on the simulator I would get can't allocate region error after about 1GB of the download.

So then I tried it on the device. Now it seems to just crash at a random time and

no crash report is put in the device, therefor I'm having an issue debugging this problem.

At first I thought it was the device or even the simulator. But i'm not really sure.

Someone mentioned that S3 framework times out the downloads randomly occasionally for large files. Could this be the case?

I'm building the file by opening a data file seeking to the end, adding the data, then closing the file until the download is complete.

I'm not sure how to debug this problem.

Any help would be appreciated.

Thank you.

Answer 1

I am a maintainer of the AWS SDK for iOS. We recently patched the S3GetObjectResponse to allow the streaming of the data directly to disk without keeping the response data in memory.

S3GetObjectResponse.m

To enable this, you simply need to set the stream when creating your request:

NSOutputStream *outputStream = [[[NSOutputStream alloc] initToFileAtPath:FILE_NAME append:NO] autorelease];
[outputStream open];
S3GetObjectRequest *getObjectRequest = [[[S3GetObjectRequest alloc] initWithKey:FILE_NAME withBucket:BUCKET_NAME] autorelease];
getObjectRequest.outputStream = outputStream;
[s3 getObject:getObjectRequest];

Update: We added a post to our AWS Mobile Developer Blog on downloading large files with the AWS SDK for iOS that includes this info as well as other tips.

Answer 2

S3GetObjectRequest has NSMutableData* body where it appends all the data it downloads.

For large files as download progresses data is appended constantly, and it goes over the VM limit of 90MB and then app gets killed by iOS.

Quick and dirty workaround is to create your own S3GetObjectRequest and S3GetObjectResponse classes. AWS framework instantiates Response based on Class Name of Request (Class name of Request without last 7 chars "Request" and appends it with "Response", and tries to instantiate new class of that name).

Then to override -(void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)data to release body all the time.

This is quick and dirty fix simply because you still have constant data allocation, appending and then release. But it works when you are in a pinch. For my usage of downloading files of 150-700mb, this simple hack kept memory usage of the app at 2.55mb average, +/- 0.2mb.

As stated by the author of ASIHTTP library, it is no longer maintained.

Request - LargeFileS3GetObjectRequest.h

@interface LargeFileS3GetObjectRequest : S3GetObjectRequest
@end

Request - LargeFileS3GetObjectRequest.m

@implementation LargeFileS3GetObjectRequest
@end

Response - LargeFileS3GetObjectResponse.h

@interface LargeFileS3GetObjectResponse : S3GetObjectResponse
@end

Response - LargeFileS3GetObjectResponse.m

@implementation LargeFileS3GetObjectResponse

-(void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)data
{
    // allow original implementation to send data to delegates
    [super connection:connection didReceiveData:data];

    // release body and set it to NULL so that underlying implementation doesn't
    // append on released object, but instead allocates new one
    [body release];
    body = NULL;
}
@end

Hope it helps.