NSURLSession and amazon S3 uploads

Question

I have an app which is currently uploading images to amazon S3. I have been trying to switch it from using NSURLConnection to NSURLSession so that the uploads can continue while the app is in the background! I seem to be hitting a bit of an issue. The NSURLRequest is created and passed to the NSURLSession but amazon sends back a 403 - forbidden response, if I pass the same request to a NSURLConnection it uploads the file perfectly.

Here is the code that creates the response:

NSString *requestURLString = [NSString stringWithFormat:@"http://%@.%@/%@/%@", BUCKET_NAME, AWS_HOST, DIRECTORY_NAME, filename];
NSURL *requestURL = [NSURL URLWithString:requestURLString];
NSMutableURLRequest *request = [NSMutableURLRequest requestWithURL:requestURL
                                                       cachePolicy:NSURLRequestReloadIgnoringLocalAndRemoteCacheData
                                                   timeoutInterval:60.0];
// Configure request
[request setHTTPMethod:@"PUT"];
[request setValue:[NSString stringWithFormat:@"%@.%@", BUCKET_NAME, AWS_HOST] forHTTPHeaderField:@"Host"];
[request setValue:[self formattedDateString] forHTTPHeaderField:@"Date"];
[request setValue:@"public-read" forHTTPHeaderField:@"x-amz-acl"];
[request setHTTPBody:imageData];

And then this signs the response (I think this came from another SO answer):

NSString *contentMd5  = [request valueForHTTPHeaderField:@"Content-MD5"];
NSString *contentType = [request valueForHTTPHeaderField:@"Content-Type"];
NSString *timestamp   = [request valueForHTTPHeaderField:@"Date"];

if (nil == contentMd5)  contentMd5  = @"";
if (nil == contentType) contentType = @"";

NSMutableString *canonicalizedAmzHeaders = [NSMutableString string];

NSArray *sortedHeaders = [[[request allHTTPHeaderFields] allKeys] sortedArrayUsingSelector:@selector(caseInsensitiveCompare:)];

for (id key in sortedHeaders)
{
    NSString *keyName = [(NSString *)key lowercaseString];
    if ([keyName hasPrefix:@"x-amz-"]){
        [canonicalizedAmzHeaders appendFormat:@"%@:%@\n", keyName, [request valueForHTTPHeaderField:(NSString *)key]];
    }
}

NSString *bucket = @"";
NSString *path   = request.URL.path;
NSString *query  = request.URL.query;

NSString *host  = [request valueForHTTPHeaderField:@"Host"];

if (![host isEqualToString:@"s3.amazonaws.com"]) {
    bucket = [host substringToIndex:[host rangeOfString:@".s3.amazonaws.com"].location];
}

NSString* canonicalizedResource;

if (nil == path || path.length < 1) {
    if ( nil == bucket || bucket.length < 1 ) {
        canonicalizedResource = @"/";
    }
    else {
        canonicalizedResource = [NSString stringWithFormat:@"/%@/", bucket];
    }
}
else {
    canonicalizedResource = [NSString stringWithFormat:@"/%@%@", bucket, path];
}

if (query != nil && [query length] > 0) {
    canonicalizedResource = [canonicalizedResource stringByAppendingFormat:@"?%@", query];
}

NSString* stringToSign = [NSString stringWithFormat:@"%@\n%@\n%@\n%@\n%@%@", [request HTTPMethod], contentMd5, contentType, timestamp, canonicalizedAmzHeaders, canonicalizedResource];

NSString *signature = [self signatureForString:stringToSign];

[request setValue:[NSString stringWithFormat:@"AWS %@:%@", self.S3AccessKey, signature] forHTTPHeaderField:@"Authorization"];

Then if I use this line of code:

[NSURLConnection connectionWithRequest:request delegate:self];

It works and uploads the file, but if I use:

NSURLSessionUploadTask *task = [self.session uploadTaskWithRequest:request fromFile:[NSURL fileURLWithPath:filePath]];
[task resume];

I get the forbidden error..!?

Has anyone tried uploading to S3 with this and hit similar issues? I wonder if it is to do with the way the session pauses and resumes uploads, or it is doing something funny to the request..?

One possible solution would be to upload the file to an interim server that I control and have that forward it to S3 when it is complete... but this is clearly not an ideal solution!

Any help is much appreciated!!

Thanks!

Answer 1

I made it work based on Zeev Vax answer. I want to provide some insight on problems I ran into and offer minor improvements.

Build a normal PutRequest, for instance

S3PutObjectRequest* putRequest = [[S3PutObjectRequest alloc] initWithKey:keyName inBucket:bucketName];

putRequest.credentials = credentials;
putRequest.filename = theFilePath;

Now we need to do some work the S3Client usually does for us

// set the endpoint, so it is not null
putRequest.endpoint = s3Client.endpoint;

// if you are using session based authentication, otherwise leave it out
putRequest.securityToken = messageTokenDTO.securityToken;

// sign the request (also computes md5 checksums etc.)
NSMutableURLRequest *request = [s3Client signS3Request:putRequest];

Now copy all of that to a new request. Amazon use their own NSUrlRequest class which would cause an exception

NSMutableURLRequest* request2 = [[NSMutableURLRequest alloc]initWithURL:request.URL];
[request2 setHTTPMethod:request.HTTPMethod];
[request2 setAllHTTPHeaderFields:[request allHTTPHeaderFields]];

Now we can start the actual transfer

NSURLSession* backgroundSession = [self backgroundSession];
_uploadTask = [backgroundSession uploadTaskWithRequest:request2 fromFile:[NSURL fileURLWithPath:theFilePath]];
[_uploadTask resume];

This is the code that creates the background session:

- (NSURLSession *)backgroundSession {
    static NSURLSession *session = nil;
    static dispatch_once_t onceToken;
    dispatch_once(&onceToken, ^{

        NSURLSessionConfiguration *configuration = [NSURLSessionConfiguration backgroundSessionConfiguration:@"com.example.my.unique.id"];
        session = [NSURLSession sessionWithConfiguration:configuration delegate:self delegateQueue:nil];
    });

    return session;
}

It took me a while to figure out that the session / task delegate needs to handle an auth challenge (we are in fact authentication to s3). So just implement

- (void)URLSession:(NSURLSession *)session didReceiveChallenge:(NSURLAuthenticationChallenge *)challenge completionHandler:(void (^)(NSURLSessionAuthChallengeDisposition, NSURLCredential *))completionHandler {
    NSLog(@"session did receive challenge");
    completionHandler(NSURLSessionAuthChallengePerformDefaultHandling, nil);
}

Answer 2

The answers here are slightly outdated, spent a great deal of my day trying to get this work in Swift and the new AWS SDK. So here's how to do it in Swift by using the new AWSS3PreSignedURLBuilder (available in version 2.0.7+):

class S3BackgroundUpload : NSObject {

    // Swift doesn't support static properties yet, so have to use structs to achieve the same thing.
    struct Static {
        static var session : NSURLSession?
    }

    override init() {
        super.init()

        // Note: There are probably safer ways to store the AWS credentials.
        let configPath = NSBundle.mainBundle().pathForResource("appconfig", ofType: "plist")
        let config = NSDictionary(contentsOfFile: configPath!)
        let accessKey = config.objectForKey("awsAccessKeyId") as String?
        let secretKey = config.objectForKey("awsSecretAccessKey") as String?
        let credentialsProvider = AWSStaticCredentialsProvider .credentialsWithAccessKey(accessKey!, secretKey: secretKey!)

        // AWSRegionType.USEast1 is the default S3 endpoint (use it if you don't need specific endpoints such as s3-us-west-2.amazonaws.com)
        let configuration = AWSServiceConfiguration(region: AWSRegionType.USEast1, credentialsProvider: credentialsProvider)

        // This is setting the configuration for all AWS services, you can also pass in this configuration to the AWSS3PreSignedURLBuilder directly.
        AWSServiceManager.defaultServiceManager().setDefaultServiceConfiguration(configuration)

        if Static.session == nil {
            let configIdentifier = "com.example.s3-background-upload"

            var config : NSURLSessionConfiguration
            if NSURLSessionConfiguration.respondsToSelector("backgroundSessionConfigurationWithIdentifier:") {
                // iOS8
                config = NSURLSessionConfiguration.backgroundSessionConfigurationWithIdentifier(configIdentifier)
            } else {
                // iOS7
                config = NSURLSessionConfiguration.backgroundSessionConfiguration(configIdentifier)
            }

            // NSURLSession background sessions *need* to have a delegate.
            Static.session = NSURLSession(configuration: config, delegate: self, delegateQueue: nil)
        }
    }

    func upload() {
        let s3path = "/some/path/some_file.jpg"
        let filePath = "/var/etc/etc/some_file.jpg"

        // Check if the file actually exists to prevent weird uncaught obj-c exceptions.
        if NSFileManager.defaultManager().fileExistsAtPath(filePath) == false {
            NSLog("file does not exist at %@", filePath)
            return
        }

        // NSURLSession needs the filepath in a "file://" NSURL format.
        let fileUrl = NSURL(string: "file://\(filePath)")

        let preSignedReq = AWSS3GetPreSignedURLRequest()
        preSignedReq.bucket = "bucket-name"
        preSignedReq.key = s3path
        preSignedReq.HTTPMethod = AWSHTTPMethod.PUT                   // required
        preSignedReq.contentType = "image/jpeg"                       // required
        preSignedReq.expires = NSDate(timeIntervalSinceNow: 60*60)    // required

        // The defaultS3PreSignedURLBuilder uses the global config, as specified in the init method.
        let urlBuilder = AWSS3PreSignedURLBuilder.defaultS3PreSignedURLBuilder()

        // The new AWS SDK uses BFTasks to chain requests together:
        urlBuilder.getPreSignedURL(preSignedReq).continueWithBlock { (task) -> AnyObject! in

            if task.error != nil {
                NSLog("getPreSignedURL error: %@", task.error)
                return nil
            }

            var preSignedUrl = task.result as NSURL
            NSLog("preSignedUrl: %@", preSignedUrl)

            var request = NSMutableURLRequest(URL: preSignedUrl)
            request.cachePolicy = NSURLRequestCachePolicy.ReloadIgnoringLocalCacheData

            // Make sure the content-type and http method are the same as in preSignedReq
            request.HTTPMethod = "PUT"
            request.setValue(preSignedReq.contentType, forHTTPHeaderField: "Content-Type")

            // NSURLSession background session does *not* support completionHandler, so don't set it.
            let uploadTask = Static.session?.uploadTaskWithRequest(request, fromFile: fileUrl)

            // Start the upload task:
            uploadTask?.resume()

            return nil
        }
    }
}

extension S3BackgroundUpload : NSURLSessionDelegate {

    func URLSession(session: NSURLSession, dataTask: NSURLSessionDataTask, didReceiveData data: NSData) {
        NSLog("did receive data: %@", NSString(data: data, encoding: NSUTF8StringEncoding))
    }

    func URLSession(session: NSURLSession, task: NSURLSessionTask, didCompleteWithError error: NSError?) {
        NSLog("session did complete")
        if error != nil {
            NSLog("error: %@", error!.localizedDescription)
        }
        // Finish up your post-upload tasks.
    }
}