Resume NSUrlSession on iOS10

Question

iOS 10 is going to be released soon so it worth to test applications for compatibility with it. During such test we've discovered that our app can't resume background downloads on iOS10. Code that worked well on previous versions does not work on new one, both on an emulator and on a device.

Instead of reducing our code to minimal working test case I've searched internet for NSUrlSession tutorials and tested them. Behaviour is the same: resuming works on previos versions of iOS but breaks on 10th.

Steps to Reproduce:

1. Download a project form NSUrlSession tutorial https://www.raywenderlich.com/110458/nsurlsession-tutorial-getting-started
2. Direct link: http://www.raywenderlich.com/wp-content/uploads/2016/01/HalfTunes-Final.zip
3. Build it and launch under iOS 10. Search something, for example "swift". Start a download and then hit a "Pause" and then "Resume"

Expected Results:

Download is resumed. You can check how it works with versions prior to iOS10.

Actual Results:

Download fails. In xcode console you can see:

2016-09-02 16:11:24.913 HalfTunes[35205:2279228] *** -[NSKeyedUnarchiver initForReadingWithData:]: data is NULL
2016-09-02 16:11:24.913 HalfTunes[35205:2279228] *** -[NSKeyedUnarchiver initForReadingWithData:]: data is NULL
2016-09-02 16:11:24.913 HalfTunes[35205:2279228] Invalid resume data for background download. Background downloads must use http or https and must download to an accessible file.

More scenarios:

If you activate offline mode while a file is beeng downloaded you get

Url session completed with error: Error Domain=NSURLErrorDomain Code=-1002 "unsupported URL" UserInfo={NSLocalizedDescription=unsupported URL} {
    NSLocalizedDescription = "unsupported URL";
}

when network is shut down and download never recovers when network is up again. Other use cases with pause, such as reboot, do not work either.

Additional investigation:

I've tried to check if returned resumeData is valid using code suggested in

How can I check that an NSData blob is valid as resumeData for an NSURLSessionDownloadTask?

but the target file is in place. Though resumeData format has changed and now file name is stored in NSURLSessionResumeInfoTempFileName and you have to append NSTemporaryDirectory() to it.

Beside that I've filled a bug report to apple, but they haven't replied yet.

The question (of Life, the Universe, and Everything):

Is resuming of NSUrlSession broken in all other apps? Can it be fixed on application side?

Answer 1

This problem arose from currentRequest and originalRequest NSKeyArchived encoded with an unusual root of "NSKeyedArchiveRootObjectKey" instead of NSKeyedArchiveRootObjectKey constant which is "root" literally and some other misbehaves in encoding process of NSURL(Mutable)Request.

I detected that in beta 1 and filed a bug (no. 27144153 in case you want duplicate). Even I sent an email to "Quinn the Eskimo" (eskimo1 at apple dot com) whom is support guy of NSURLSession team, to confirm they received it and he said they got that and are aware of issue.

UPDATE: I finally figured out how to resolve this issue. Give data to correctResumeData() function and it will return usable resume data

UPDATE 2: You can use NSURLSession.correctedDownloadTaskWithResumeData() / URLSession.correctedDownloadTask(withResumeData:) function to get a task with a correct originalRequest and currentRequest variables

UPDATE 3: Quinn says This issue is resolved in iOS 10.2, you can keep using this code to have compatibility with iOS 10.0 and 10.1 and it will work with new version without any issue.

(For Swift 3 code, scroll below, for Objective C see leavesstar post but I didn't tested it)

Swift 2.3:

func correctRequestData(data: NSData?) -> NSData? {
    guard let data = data else {
        return nil
    }
    // return the same data if it's correct
    if NSKeyedUnarchiver.unarchiveObjectWithData(data) != nil {
        return data
    }
    guard let archive = (try? NSPropertyListSerialization.propertyListWithData(data, options: [.MutableContainersAndLeaves], format: nil)) as? NSMutableDictionary else {
        return nil
    }
    // Rectify weird __nsurlrequest_proto_props objects to $number pattern
    var k = 0
    while archive["$objects"]?[1].objectForKey("$\(k)") != nil {
        k += 1
    }
    var i = 0
    while archive["$objects"]?[1].objectForKey("__nsurlrequest_proto_prop_obj_\(i)") != nil {
        let arr = archive["$objects"] as? NSMutableArray
        if let dic = arr?[1] as? NSMutableDictionary, let obj = dic["__nsurlrequest_proto_prop_obj_\(i)"] {
            dic.setObject(obj, forKey: "$\(i + k)")
            dic.removeObjectForKey("__nsurlrequest_proto_prop_obj_\(i)")
            arr?[1] = dic
            archive["$objects"] = arr
        }
        i += 1
    }
    if archive["$objects"]?[1].objectForKey("__nsurlrequest_proto_props") != nil {
        let arr = archive["$objects"] as? NSMutableArray
        if let dic = arr?[1] as? NSMutableDictionary, let obj = dic["__nsurlrequest_proto_props"] {
            dic.setObject(obj, forKey: "$\(i + k)")
            dic.removeObjectForKey("__nsurlrequest_proto_props")
            arr?[1] = dic
            archive["$objects"] = arr
        }
    }
    // Rectify weird "NSKeyedArchiveRootObjectKey" top key to NSKeyedArchiveRootObjectKey = "root"
    if archive["$top"]?.objectForKey("NSKeyedArchiveRootObjectKey") != nil {
        archive["$top"]?.setObject(archive["$top"]?["NSKeyedArchiveRootObjectKey"], forKey: NSKeyedArchiveRootObjectKey)
        archive["$top"]?.removeObjectForKey("NSKeyedArchiveRootObjectKey")
    }
    // Reencode archived object
    let result = try? NSPropertyListSerialization.dataWithPropertyList(archive, format: NSPropertyListFormat.BinaryFormat_v1_0, options: NSPropertyListWriteOptions())
    return result
}

func getResumeDictionary(data: NSData) -> NSMutableDictionary? {
    var iresumeDictionary: NSMutableDictionary? = nil
    // In beta versions, resumeData is NSKeyedArchive encoded instead of plist
    if #available(iOS 10.0, OSX 10.12, *) {
        var root : AnyObject? = nil
        let keyedUnarchiver = NSKeyedUnarchiver(forReadingWithData: data)

        do {
            root = try keyedUnarchiver.decodeTopLevelObjectForKey("NSKeyedArchiveRootObjectKey") ?? nil
            if root == nil {
                root = try keyedUnarchiver.decodeTopLevelObjectForKey(NSKeyedArchiveRootObjectKey)
            }
        } catch {}
        keyedUnarchiver.finishDecoding()
        iresumeDictionary = root as? NSMutableDictionary

    }

    if iresumeDictionary == nil {
        do {
            iresumeDictionary = try NSPropertyListSerialization.propertyListWithData(data, options: [.MutableContainersAndLeaves], format: nil) as? NSMutableDictionary;
        } catch {}
    }

    return iresumeDictionary
}

func correctResumeData(data: NSData?) -> NSData? {
    let kResumeCurrentRequest = "NSURLSessionResumeCurrentRequest"
    let kResumeOriginalRequest = "NSURLSessionResumeOriginalRequest"

    guard let data = data, let resumeDictionary = getResumeDictionary(data) else {
        return nil
    }

    resumeDictionary[kResumeCurrentRequest] = correctRequestData(resumeDictionary[kResumeCurrentRequest] as? NSData)
    resumeDictionary[kResumeOriginalRequest] = correctRequestData(resumeDictionary[kResumeOriginalRequest] as? NSData)

    let result = try? NSPropertyListSerialization.dataWithPropertyList(resumeDictionary, format: NSPropertyListFormat.XMLFormat_v1_0, options: NSPropertyListWriteOptions())
    return result
}

extension NSURLSession {
    func correctedDownloadTaskWithResumeData(resumeData: NSData) -> NSURLSessionDownloadTask {
        let kResumeCurrentRequest = "NSURLSessionResumeCurrentRequest"
        let kResumeOriginalRequest = "NSURLSessionResumeOriginalRequest"

        let cData = correctResumeData(resumeData) ?? resumeData
        let task = self.downloadTaskWithResumeData(cData)

        // a compensation for inability to set task requests in CFNetwork.
        // While you still get -[NSKeyedUnarchiver initForReadingWithData:]: data is NULL error,
        // this section will set them to real objects
        if let resumeDic = getResumeDictionary(cData) {
            if task.originalRequest == nil, let originalReqData = resumeDic[kResumeOriginalRequest] as? NSData, let originalRequest = NSKeyedUnarchiver.unarchiveObjectWithData(originalReqData) as? NSURLRequest {
                task.setValue(originalRequest, forKey: "originalRequest")
            }
            if task.currentRequest == nil, let currentReqData = resumeDic[kResumeCurrentRequest] as? NSData, let currentRequest = NSKeyedUnarchiver.unarchiveObjectWithData(currentReqData) as? NSURLRequest {
                task.setValue(currentRequest, forKey: "currentRequest")
            }
        }

        return task
    }
}

Swift 3:

func correct(requestData data: Data?) -> Data? {
    guard let data = data else {
        return nil
    }
    if NSKeyedUnarchiver.unarchiveObject(with: data) != nil {
        return data
    }
    guard let archive = (try? PropertyListSerialization.propertyList(from: data, options: [.mutableContainersAndLeaves], format: nil)) as? NSMutableDictionary else {
        return nil
    }
    // Rectify weird __nsurlrequest_proto_props objects to $number pattern
    var k = 0
    while ((archive["$objects"] as? NSArray)?[1] as? NSDictionary)?.object(forKey: "$\(k)") != nil {
        k += 1
    }
    var i = 0
    while ((archive["$objects"] as? NSArray)?[1] as? NSDictionary)?.object(forKey: "__nsurlrequest_proto_prop_obj_\(i)") != nil {
        let arr = archive["$objects"] as? NSMutableArray
        if let dic = arr?[1] as? NSMutableDictionary, let obj = dic["__nsurlrequest_proto_prop_obj_\(i)"] {
            dic.setObject(obj, forKey: "$\(i + k)" as NSString)
            dic.removeObject(forKey: "__nsurlrequest_proto_prop_obj_\(i)")
            arr?[1] = dic
            archive["$objects"] = arr
        }
        i += 1
    }
    if ((archive["$objects"] as? NSArray)?[1] as? NSDictionary)?.object(forKey: "__nsurlrequest_proto_props") != nil {
        let arr = archive["$objects"] as? NSMutableArray
        if let dic = arr?[1] as? NSMutableDictionary, let obj = dic["__nsurlrequest_proto_props"] {
            dic.setObject(obj, forKey: "$\(i + k)" as NSString)
            dic.removeObject(forKey: "__nsurlrequest_proto_props")
            arr?[1] = dic
            archive["$objects"] = arr
        }
    }
    /* I think we have no reason to keep this section in effect 
    for item in (archive["$objects"] as? NSMutableArray) ?? [] {
        if let cls = item as? NSMutableDictionary, cls["$classname"] as? NSString == "NSURLRequest" {
            cls["$classname"] = NSString(string: "NSMutableURLRequest")
            (cls["$classes"] as? NSMutableArray)?.insert(NSString(string: "NSMutableURLRequest"), at: 0)
        }
    }*/
    // Rectify weird "NSKeyedArchiveRootObjectKey" top key to NSKeyedArchiveRootObjectKey = "root"
    if let obj = (archive["$top"] as? NSMutableDictionary)?.object(forKey: "NSKeyedArchiveRootObjectKey") as AnyObject? {
        (archive["$top"] as? NSMutableDictionary)?.setObject(obj, forKey: NSKeyedArchiveRootObjectKey as NSString)
        (archive["$top"] as? NSMutableDictionary)?.removeObject(forKey: "NSKeyedArchiveRootObjectKey")
    }
    // Reencode archived object
    let result = try? PropertyListSerialization.data(fromPropertyList: archive, format: PropertyListSerialization.PropertyListFormat.binary, options: PropertyListSerialization.WriteOptions())
    return result
}

func getResumeDictionary(_ data: Data) -> NSMutableDictionary? {
    // In beta versions, resumeData is NSKeyedArchive encoded instead of plist
    var iresumeDictionary: NSMutableDictionary? = nil
    if #available(iOS 10.0, OSX 10.12, *) {
        var root : AnyObject? = nil
        let keyedUnarchiver = NSKeyedUnarchiver(forReadingWith: data)

        do {
            root = try keyedUnarchiver.decodeTopLevelObject(forKey: "NSKeyedArchiveRootObjectKey") ?? nil
            if root == nil {
                root = try keyedUnarchiver.decodeTopLevelObject(forKey: NSKeyedArchiveRootObjectKey)
            }
        } catch {}
        keyedUnarchiver.finishDecoding()
        iresumeDictionary = root as? NSMutableDictionary

    }

    if iresumeDictionary == nil {
        do {
            iresumeDictionary = try PropertyListSerialization.propertyList(from: data, options: PropertyListSerialization.ReadOptions(), format: nil) as? NSMutableDictionary;
        } catch {}
    }

    return iresumeDictionary
}

func correctResumeData(_ data: Data?) -> Data? {
    let kResumeCurrentRequest = "NSURLSessionResumeCurrentRequest"
    let kResumeOriginalRequest = "NSURLSessionResumeOriginalRequest"

    guard let data = data, let resumeDictionary = getResumeDictionary(data) else {
        return nil
    }

    resumeDictionary[kResumeCurrentRequest] = correct(requestData: resumeDictionary[kResumeCurrentRequest] as? Data)
    resumeDictionary[kResumeOriginalRequest] = correct(requestData: resumeDictionary[kResumeOriginalRequest] as? Data)

    let result = try? PropertyListSerialization.data(fromPropertyList: resumeDictionary, format: PropertyListSerialization.PropertyListFormat.xml, options: PropertyListSerialization.WriteOptions())
    return result
}


extension URLSession {
    func correctedDownloadTask(withResumeData resumeData: Data) -> URLSessionDownloadTask {
        let kResumeCurrentRequest = "NSURLSessionResumeCurrentRequest"
        let kResumeOriginalRequest = "NSURLSessionResumeOriginalRequest"

        let cData = correctResumeData(resumeData) ?? resumeData
        let task = self.downloadTask(withResumeData: cData)

        // a compensation for inability to set task requests in CFNetwork.
        // While you still get -[NSKeyedUnarchiver initForReadingWithData:]: data is NULL error,
        // this section will set them to real objects
        if let resumeDic = getResumeDictionary(cData) {
            if task.originalRequest == nil, let originalReqData = resumeDic[kResumeOriginalRequest] as? Data, let originalRequest = NSKeyedUnarchiver.unarchiveObject(with: originalReqData) as? NSURLRequest {
                task.setValue(originalRequest, forKey: "originalRequest")
            }
            if task.currentRequest == nil, let currentReqData = resumeDic[kResumeCurrentRequest] as? Data, let currentRequest = NSKeyedUnarchiver.unarchiveObject(with: currentReqData) as? NSURLRequest {
                task.setValue(currentRequest, forKey: "currentRequest")
            }
        }

        return task
    }
}

Answer 2

Regarding the part of the question about the unsupported URL error and lost resumeData on network fail, or other fail, I've logged a TSI with Apple, and the latest response from Quinn:

Firstly, the behaviour you’re seeing is definitely a bug in NSURLSession. We hope to fix this problem in a future software update. That work is being tracked by . I don’t have any info to share as to when the fix will ship to normal iOS users.

As to workarounds, I dug into this issue in detail yesterday and I now fully understand the failure. IMO there is a reasonable way to work around this but I need to run my ideas past NSURLSession engineering before I can share them. I hope to hear back from them in the next day or two. Please stand by.

I'll post updates as they happen, but Im sure this gives people some hope that at least the issue is being looked at by apple.

(Massive props to Mousavian workaround for suspend/resume behaviour)

UPDATE:

From Quinn,

Indeed. Since we last talked (and I apologise that I’ve taken so long to get back to you here; I’ve been buried in incidents recently) I’ve dug into this further on behalf of some other developers and discovered that: A. This issue manifests in two contexts, characterised by the NSURLErrorCannotWriteToFile and NSURLErrorUnsupportedURL errors. B. We can work around the first but not the second. I’ve attached an update to my doc that fills in the details. Unfortunately we were unable to come up with a workaround for the second symptom. The only way forward is for iOS Engineering to fix that bug. We hope that will happen in an iOS 10 software update but I don’t have any concrete details to share (other than that this fix looks like it missed the 10.1 bus)-:

So, unfortunately, the unsupported URL issue has no work around and we have to wait for the bug to be fixed.

The NSURLErrorCannotWriteToFile issue is handled by Mousavian's code above.

ANOTHER UPDATE:

Quinn confirmed the latest 10.2 beta attempts to resolve these issues.

Did this get a look in on 10.2?

Yes. The fix for this problem was included in the first 10.2 beta. A number of developers I’ve worked with have reported that this patch has stuck, but I still recommend that you try it for yourself on the latest beta (currently iOS 10.2 beta 2, 14C5069c). Let me know if you hit any snags.

Answer 3

Here is the Objective - C code for Mousavian's answer.

It works fine in iOS 9.3.5(Device) and iOS 10.1 (simulator).

First correct the Resume data in Mousavian's way

 - (NSData *)correctRequestData:(NSData *)data
{
    if (!data) {
        return nil;
    }
    if ([NSKeyedUnarchiver unarchiveObjectWithData:data]) {
        return data;
    }

    NSMutableDictionary *archive = [NSPropertyListSerialization propertyListWithData:data options:NSPropertyListMutableContainersAndLeaves format:nil error:nil];
    if (!archive) {
        return nil;
    }
    int k = 0;
    while ([[archive[@"$objects"] objectAtIndex:1] objectForKey:[NSString stringWithFormat:@"$%d", k]]) {
        k += 1;
    }

    int i = 0;
    while ([[archive[@"$objects"] objectAtIndex:1] objectForKey:[NSString stringWithFormat:@"__nsurlrequest_proto_prop_obj_%d", i]]) {
        NSMutableArray *arr = archive[@"$objects"];
        NSMutableDictionary *dic = [arr objectAtIndex:1];
        id obj;
        if (dic) {
            obj = [dic objectForKey:[NSString stringWithFormat:@"__nsurlrequest_proto_prop_obj_%d", i]];
            if (obj) {
                [dic setObject:obj forKey:[NSString stringWithFormat:@"$%d",i + k]];
                [dic removeObjectForKey:[NSString stringWithFormat:@"__nsurlrequest_proto_prop_obj_%d", i]];
                arr[1] = dic;
                archive[@"$objects"] = arr;
            }
        }
        i += 1;
    }
    if ([[archive[@"$objects"] objectAtIndex:1] objectForKey:@"__nsurlrequest_proto_props"]) {
        NSMutableArray *arr = archive[@"$objects"];
        NSMutableDictionary *dic = [arr objectAtIndex:1];
        if (dic) {
            id obj;
            obj = [dic objectForKey:@"__nsurlrequest_proto_props"];
            if (obj) {
                [dic setObject:obj forKey:[NSString stringWithFormat:@"$%d",i + k]];
                [dic removeObjectForKey:@"__nsurlrequest_proto_props"];
                arr[1] = dic;
                archive[@"$objects"] = arr;
            }
        }
    }

    id obj = [archive[@"$top"] objectForKey:@"NSKeyedArchiveRootObjectKey"];
    if (obj) {
        [archive[@"$top"] setObject:obj forKey:NSKeyedArchiveRootObjectKey];
        [archive[@"$top"] removeObjectForKey:@"NSKeyedArchiveRootObjectKey"];
    }
    NSData *result = [NSPropertyListSerialization dataWithPropertyList:archive format:NSPropertyListBinaryFormat_v1_0 options:0 error:nil];
    return result;
}

- (NSMutableDictionary *)getResumDictionary:(NSData *)data
{
    NSMutableDictionary *iresumeDictionary;
    if ([[NSProcessInfo processInfo] operatingSystemVersion].majorVersion >= 10) {
        NSMutableDictionary *root;
        NSKeyedUnarchiver *keyedUnarchiver = [[NSKeyedUnarchiver alloc] initForReadingWithData:data];
        NSError *error = nil;
        root = [keyedUnarchiver decodeTopLevelObjectForKey:@"NSKeyedArchiveRootObjectKey" error:&error];
        if (!root) {
            root = [keyedUnarchiver decodeTopLevelObjectForKey:NSKeyedArchiveRootObjectKey error:&error];
        }
        [keyedUnarchiver finishDecoding];
        iresumeDictionary = root;
    }

    if (!iresumeDictionary) {
        iresumeDictionary = [NSPropertyListSerialization propertyListWithData:data options:0 format:nil error:nil];
    }
    return iresumeDictionary;
}

static NSString * kResumeCurrentRequest = @"NSURLSessionResumeCurrentRequest";
static NSString * kResumeOriginalRequest = @"NSURLSessionResumeOriginalRequest";
- (NSData *)correctResumData:(NSData *)data
{
    NSMutableDictionary *resumeDictionary = [self getResumDictionary:data];
    if (!data || !resumeDictionary) {
        return nil;
    }

    resumeDictionary[kResumeCurrentRequest] = [self correctRequestData:[resumeDictionary objectForKey:kResumeCurrentRequest]];
    resumeDictionary[kResumeOriginalRequest] = [self correctRequestData:[resumeDictionary objectForKey:kResumeOriginalRequest]];

    NSData *result = [NSPropertyListSerialization dataWithPropertyList:resumeDictionary format:NSPropertyListXMLFormat_v1_0 options:0 error:nil];
    return result;
}

I didn't create a category for NSURLSession, I just create in My Singleton. here is code for create NSURLSessionDownloadTask:

    NSData *cData = [self correctResumData:self.resumeData];
    if (!cData) {
        cData = self.resumeData;
    }
    self.downloadTask = [self.session downloadTaskWithResumeData:cData];
    if ([self getResumDictionary:cData]) {
        NSDictionary *dict = [self getResumDictionary:cData];
        if (!self.downloadTask.originalRequest) {
            NSData *originalData = dict[kResumeOriginalRequest];
            [self.downloadTask setValue:[NSKeyedUnarchiver unarchiveObjectWithData:originalData] forKey:@"originalRequest"];
        }
        if (!self.downloadTask.currentRequest) {
            NSData *currentData = dict[kResumeCurrentRequest];
            [self.downloadTask setValue:[NSKeyedUnarchiver unarchiveObjectWithData:currentData] forKey:@"currentRequest"];
        }
    }