Efficient way to upload multiple images to S3 from iOS

Question

I'm using Amazon S3 as my file storage system in the app. All my item objects have several images associated with them, and each stores only the image urls to keep my database lightweight. As such I need an efficient way to upload multiple images to S3 directly from iOS, and upon successful completion store their urls in the object I send to the server. I've perused the SDK and sample app Amazon provides, but the only example I've come across is for a single image upload, and goes as follows:

 func uploadData(data: NSData) {
    let expression = AWSS3TransferUtilityUploadExpression()
    expression.progressBlock = progressBlock

    let transferUtility = AWSS3TransferUtility.defaultS3TransferUtility()

    transferUtility.uploadData(
        data,
        bucket: S3BucketName,
        key: S3UploadKeyName,
        contentType: "text/plain",
        expression: expression,
        completionHander: completionHandler).continueWithBlock { (task) -> AnyObject! in
            if let error = task.error {
                NSLog("Error: %@",error.localizedDescription);
                self.statusLabel.text = "Failed"
            }
            if let exception = task.exception {
                NSLog("Exception: %@",exception.description);
                self.statusLabel.text = "Failed"
            }
            if let _ = task.result {
                self.statusLabel.text = "Generating Upload File"
                NSLog("Upload Starting!")
                // Do something with uploadTask.
            }

            return nil;
    }
}

For upwards of 5 images this will become a nested mess since I'd have to wait for each upload to return successfully before initiating the next, and then finally sending the object to my DB. Is there an efficient, code-clean for me to accomplish my goal?

URL to Amazon's sample app github: https://github.com/awslabs/aws-sdk-ios-samples/tree/master/S3TransferUtility-Sample/Swift

Answer 1

Here is my code I used to upload multiple images to S3, in the same time using DispatchGroup().

func uploadOfferImagesToS3() {
    let group = DispatchGroup()

    for (index, image) in arrOfImages.enumerated() {
        group.enter()

        Utils.saveImageToTemporaryDirectory(image: image, completionHandler: { (url, imgScalled) in
            if let urlImagePath = url,
                let uploadRequest = AWSS3TransferManagerUploadRequest() {
                uploadRequest.body          = urlImagePath
                uploadRequest.key           = ProcessInfo.processInfo.globallyUniqueString + "." + "png"
                uploadRequest.bucket        = Constants.AWS_S3.Image
                uploadRequest.contentType   = "image/" + "png"
                uploadRequest.uploadProgress = {(bytesSent:Int64, totalBytesSent:Int64, totalBytesExpectedToSend:Int64) in
                    let uploadProgress = Float(Double(totalBytesSent)/Double(totalBytesExpectedToSend))

                    print("uploading image \(index) of \(arrOfImages.count) = \(uploadProgress)")

                    //self.delegate?.amazonManager_uploadWithProgress(fProgress: uploadProgress)
                }

                self.uploadImageStatus      = .inProgress

                AWSS3TransferManager.default()
                    .upload(uploadRequest)
                    .continueWith(executor: AWSExecutor.immediate(), block: { (task) -> Any? in

                        group.leave()

                        if let error = task.error {
                            print("\n\n=======================================")
                            print("❌ Upload image failed with error: (\(error.localizedDescription))")
                            print("=======================================\n\n")

                            self.uploadImageStatus = .failed
                            self.delegate?.amazonManager_uploadWithFail()

                            return nil
                        }

                        //=>    Task completed successfully
                        let imgS3URL = Constants.AWS_S3.BucketPath + Constants.AWS_S3.Image + "/" + uploadRequest.key!
                        print("imgS3url = \(imgS3URL)")
                        NewOfferManager.shared.arrUrlsImagesNewOffer.append(imgS3URL)

                        self.uploadImageStatus = .completed
                        self.delegate?.amazonManager_uploadWithSuccess(strS3ObjUrl: imgS3URL, imgSelected: imgScalled)

                        return nil
                    })
            }
            else {
                print(" Unable to save image to NSTemporaryDirectory")
            }
        })
    }

    group.notify(queue: DispatchQueue.global(qos: .background)) {
        print("All \(arrOfImages.count) network reqeusts completed")
    }
}

and this is the critical part, where I lost at least 5 hours. URLs from NSTemporaryDirectory must be different for each image!!!

class func saveImageToTemporaryDirectory(image: UIImage, completionHandler: @escaping (_ url: URL?, _ imgScalled: UIImage) -> Void) {
    let imgScalled              = ClaimitUtils.scaleImageDown(image)
    let data                    = UIImagePNGRepresentation(imgScalled)

    let randomPath = "offerImage" + String.random(ofLength: 5)

    let urlImgOfferDir = URL(fileURLWithPath: NSTemporaryDirectory().appending(randomPath))
    do {
        try data?.write(to: urlImgOfferDir)
        completionHandler(urlImgOfferDir, imgScalled)
    }
    catch (let error) {
        print(error)
        completionHandler(nil, imgScalled)
    }
}

Hope this will help!

Answer 2

As I said on H. Al-Amri's response, if you need to know when the last upload is complete you can't simply iterate through an array of data and upload them all at once.

In Javascript there is a library (Async.js I think) that will make it easy to do background operations on individual elements of an array, and to get callbacks when each one finishes and when the entire array is finished. Since I'm not aware of anything like that for Swift, your intuition that you have to chain the uploads is correct.

As @DavidTamrazov explained in comments, you can chain your calls together using recursion. The way I solved this problem was a little more complicated since my network operations are done using NSOperationQueue to chain NSOperations. I pass an array of images to a custom NSOperation that uploads the first image from the array. When it completes, it adds another NSOperation to my NSOperationsQueue with the array of remaining images. When the array runs out of images, I know I'm complete.

Following is an example I chopped out of a much much larger chunk I use. Treat it as pseudocode because I edited it heavily and didn't have time to even compile it. But hopefully it's clear enough on the framework for how to do this with NSOperations.

class NetworkOp : Operation {
    var isRunning = false

    override var isAsynchronous: Bool {
        get {
            return true
        }
    }

    override var isConcurrent: Bool {
        get {
            return true
        }
    }

    override var isExecuting: Bool {
        get {
            return isRunning
        }
    }

    override var isFinished: Bool {
        get {
            return !isRunning
        }
    }

    override func start() {
        if self.checkCancel() {
            return
        }
        self.willChangeValue(forKey: "isExecuting")
        self.isRunning = true
        self.didChangeValue(forKey: "isExecuting")
        main()
    }

    func complete() {
        self.willChangeValue(forKey: "isFinished")
        self.willChangeValue(forKey: "isExecuting")
        self.isRunning = false
        self.didChangeValue(forKey: "isFinished")
        self.didChangeValue(forKey: "isExecuting")
        print( "Completed net op: \(self.className)")
    }

    // Always resubmit if we get canceled before completion
    func checkCancel() -> Bool {
        if self.isCancelled {
            self.retry()
            self.complete()
        }
        return self.isCancelled
    }

    func retry() {
        // Create a new NetworkOp to match and resubmit since we can't reuse existing.
    }

    func success() {
        // Success means reset delay
        NetOpsQueueMgr.shared.resetRetryIncrement()
    }
}

class ImagesUploadOp : NetworkOp {
    var imageList : [PhotoFileListMap]

    init(imageList : [UIImage]) {
        self.imageList = imageList
    }

    override func main() {
        print( "Photos upload starting")
        if self.checkCancel() {
            return
        }

        // Pop image off front of array
        let image = imageList.remove(at: 0)

        // Now call function that uses AWS to upload image, mine does save to file first, then passes
        // an error message on completion if it failed, nil if it succceeded
        ServerMgr.shared.uploadImage(image: image, completion: {  errorMessage ) in
            if let error = errorMessage {
                print("Failed to upload file - " + error)
                self.retry()
            } else {
                print("Uploaded file")
                if !self.isCancelled {
                    if self.imageList.count == 0 {
                        // All images done, here you could call a final completion handler or somthing.
                    } else {
                        // More images left to do, let's put another Operation on the barbie:)
                        NetOpsQueueMgr.shared.submitOp(netOp: ImagesUploadOp(imageList: self.imageList))
                    }
                }
            }
            self.complete()
         })
    }

    override func retry() {
        NetOpsQueueMgr.shared.retryOpWithDelay(op: ImagesUploadOp(form: self.form, imageList: self.imageList))
    }
}


// MARK: NetOpsQueueMgr  -------------------------------------------------------------------------------

class NetOpsQueueMgr {
    static let shared = NetOpsQueueMgr()

    lazy var opsQueue :OperationQueue = {
        var queue = OperationQueue()
        queue.name = "myQueName"
        queue.maxConcurrentOperationCount = 1
        return queue
    }()

    func submitOp(netOp : NetworkOp) {
         opsQueue.addOperation(netOp)
    }

   func uploadImages(imageList : [UIImage]) {
        let imagesOp = ImagesUploadOp(form: form, imageList: imageList)
        self.submitOp(netOp: imagesOp)
    }
}