Migrating data when iCloud is turned on/off

Question

The local account

From the WWDC 2013 207 session about Core Data and iCloud:

You provide us a single store URL inside the application's local sandbox and we then create an opaque container with an entry inside of it for each account on the system, including the local account, which is our term for what happens when there is no iCloud account on the system. This is a special store that's managed by Core Data so that you don't have to do anything special because your user doesn't have an iCloud account.

In iOS 7/OS X 10.9, Core Data with iCloud will automatically use a local account for situations in which iCloud is off. Unlike the fallback store (used when iCloud is on but unreachable), the local account will be wholly replaced by an iCloud account when the service is on, without any merging. The data in the local account is only accesible if iCloud is off. This happens when:

• There is no iCloud account.
• There is an iCloud account, but "Documents & Data" has been disabled.
• There is an iCloud account, but the app has been disabled in "Documents & Data".

The above is what I understand from experimentation. Please correct me if I'm wrong.

When data disappears

Used as is, the local account user experience is awful. If you add data to an app with iCloud off and then turn it on, the data will "disappear" and you might think that it has been deleted. If you add data to an app with iCloud on, and then turn it off, the data will also "disappear".

I have seen examples that try to work around this by adding (more) iCloud settings to the app and managing their own "local" store (not the one provided by iCloud). This reeks of duplicating work to me.

Leveraging the local account for data migration

How about this approach?

• Always use Core Data and iCloud, no matter if iCloud is on or off.
• When iCloud goes from off to on, ask users if they want to merge the local account with the iCloud account. If yes, merge, remove duplicates prioritizing local and empty the local account.
• When iCloud goes from on to off, ask users if they want to merge the iCloud store with the local account. If yes, merge and remove duplicates prioritizing iCloud.

This is similar to what Reminders does. However, Reminders asks the user about data migration directly from iCloud settings, which is something that us developers can't do.

Questions

1) Does this approach have any drawbacks or border cases that might not be obvious at first glance? Maybe we're not meant to use the iCloud-generated local account like this.

2) Are NSPersistentStoreCoordinatorStoresWillChangeNotification and NSPersistentStoreCoordinatorStoresDidChangeNotification sufficient to detect all the possible on to off and off to on iCloud transitions?

3) Would you do the user prompt and merging between NSPersistentStoreCoordinatorStoresWillChangeNotification and NSPersistentStoreCoordinatorStoresDidChangeNotification, or gather all the information in those and wait until the store is changed? I ask because these notifications appear to be sent in background, and blocking them to perform a potentially long operation might not be what Core Data expects.

Answer 1

I think you have misunderstood what was said in the 207 session.

Core Data will not automatically create a local and an iCloud store for you, well not ones that will synchronise data when the iCloud account if turned off anyway. Depending on what the user has selected you have to create the store either using the NSPersistentStoreUbiquityNameKey option (for an iCloud store) or not using it (for a local store).

Because the default security setting for a new apps Data&Documents is ON when your app is first installed you MUST ask the user if they want to use iCloud or not. Try it out with Apple's Pages app.

If the user subsequently changes the preference setting your App must migrate the store to or from iCloud.

The part Core Data handles automatically is if you switch the iCloud Account (log out and log in with a different account) then the App will run with whatever Core Data store might have been created while logged in to this account.

See the transcript below where it quite clearly states that the iCloud store gets removed when the account goes away. It's gone, kaput, a dead parrot. So while you get a chance to save only the change logs remain locally in case the account gets used again in future.

You simply implement your will change handlers and respond to NSPersistentStoreCoordinator Stores Will Change and will notify you automatically when we need to change the persistent store file because there's new account on the system.

Of course, you can then call NSManagedObjectContext save and NSManagedObjectContext reset.

Now once you've done that, we'll remove the store from the coordinator just as with the asynchronous setup process and then we'll send you NSPersistentStoreCoordinator Storage Did Change notification, again, just like asynchronous setup and you can begin working with your application as you normally would.

Now, let's talk about this in a little bit more detail.

When you receive NSPersistentStoreCoordinator Stores Will Change notification, the persistent store is still available to use, and so unlike what we advised you of last year where you had to immediately drop the persistent store and wipe out your managed object context, you can still write to the managed object context and those changes will be persistent locally to be imported to the account if it every comes back.

This means that although your user's changes won't make it to iCloud immediately, if they ever sign in again, they'll be there and waiting.

Finally, all of these store files will be managed by Core Data and that means that we could remove them at any time.

Each store will be removed once its account has gone away because we can rebuild the file from the cloud.

So we want to free up as much disk space as possible for your application to use and not have old store files lying around that could take up additional resources.

and a bit further on

We're also introducing a new option to help you create backups or local copies of the iCloud persistent store called NSPersistentStore Remove Ubiquitous Metadata Option.

This removes all associated metadata from the iCloud store; that means, anything that we write into the metadata dictionary as well as the store file itself, and it's critical if you want to use the migration API to create backups or local copies at a persistent store you wish to open without the iCloud options.

Also take a look at this link to the errata for Tim Roadley's book

http://timroadley.com/2014/02/13/learning-core-data-for-ios-errata/

If you are logged in to iCloud and then the user changes the app preference setting (not the same as the Data&Documents security setting) to turn iCloud off your App should then ask the user if they want to migrate the existing iCloud store to the local (again - try this with Pages and see what messages you get).

I have posted a sample app that does all of this here. Take a look at the video to see the expected behaviour. http://ossh.com.au/design-and-technology/software-development/

Some of the features of the sample apps include:

Features include:

• Sample iOS and OSX Core Data Apps with iCloud Integration
• Use of Local or iCloud Core Data store
• Includes a Settings Bundle (note that this creates a settings page in the Settings App) that includes:
  • Use iCloud preference setting (ON or OFF)
  • Make Backup preference setting (ON or OFF)
  • Display application Version and Build Number
• Prompts the user about storage options when the Use iCloud preference is changed to ON
• Migrates Core Data store to and from iCloud depending on the users preference setting and response to prompts
• Detects deletion of iCloud store from another device and cleans up by creating a new empty iCloud store
• Checks for existing iCloud files when migrating local store to iCloud and prompts user whether to merge or discard data in local store if an iCloud file exists
• Makes a Backup of the Core Data store if Make Backup preference is set to ON.  Backup file name is persistentStore_Backup_yyyy_MM_dd_HH_mm_ss. To use it:
  • set Backup preference ON and next time the app is activated it will make a backup of the current Core Data store and reset the preference to OFF
  • file can be copied to PC or Mac from iTunes
  • to restore simply set app to use Local files (Use iCloud preference OFF) and replace the persistentStore file with the required backup file (note the file must be called persistentStore).
• Editing record and save/cancel edits in detailed view
• Asynchronous opening of Core Data store to ensure long migrations don't block the main thread and cause App to be terminated
• Loading of data on background thread with Pull to Refresh in main UITableView to start another background thread (you can start multiple background threads running simultaneously, take care!) 
• Display related objects in detailView using UITableView, fetchedResultsController and predicate to filter selection
• Load Seed Data if a no store exists already, checks if iCloud file has been created by another device
• iCloud Upload/Download Status indicator, network activity indicator turns on when Core Data transaction logs need to be synced, are busy syncing, being imported or when background tasks are running
• Sidebar style UI with multiple master and detail views for both iOS and OS X apps
• Backup File Manager which allows you to make backups, copy backup files to and from iCloud, send and receive backup files via email and restore from a backup file.

Answer 2

I will attempt to answer my own question, partly to organise my thoughts and partly to reply to @DuncanGroenewald.

1) Does this approach have any drawbacks or border cases that might not be obvious at first glance?

Yes. The local and iCloud account stores are managed by Core Data and can be removed at any time.

In practice, I don't think the local account store will be ever removed as it cannot be recreated from iCloud.

Regarding iCloud account stores, I can see two scenarios in which they might be removed: a) to free space after the user turned iCloud off or b) because the user requested it by selecting Settings > iCloud > Delete All.

If the user requested it, then you might argue that data migration is not a concern.

If if was to free space, then yes, it's a problem. However, the same problem exists in any other method as your app is not woken up when iCloud account stores are removed.

2) Are NSPersistentStoreCoordinatorStoresWillChangeNotification and NSPersistentStoreCoordinatorStoresDidChangeNotification sufficient to detect all the possible on to off and off to on iCloud transitions?

Yes. It requires you to always create the persistent store with NSPersistentStoreUbiquitousContentNameKey, no matter if iCloud is on or off. Like this:

[self.managedObjectContext.persistentStoreCoordinator
 addPersistentStoreWithType:NSSQLiteStoreType
 configuration:nil
 URL:storeURL
 options:@{ NSPersistentStoreUbiquitousContentNameKey : @"someName" }
 error:&error];

In fact, only listening to NSPersistentStoreCoordinatorStoresDidChangeNotification is enough (as shown below). This will be called when the store is added at startup or changed during execution.

3) Would you do the user prompt and merging between NSPersistentStoreCoordinatorStoresWillChangeNotification and NSPersistentStoreCoordinatorStoresDidChangeNotification, or gather all the information in those and wait until the store is changed? I ask because these notifications appear to be sent in background, and blocking them to perform a potentially long operation might not be what Core Data expects.

This is how I would do it in NSPersistentStoreCoordinatorStoresDidChangeNotification.

Since this notification is sent both at startup and when the store changes during execution, we can use it to save the current store url and ubiquity identity token (if any).

Then we check if we are in a on/off transition scenario and migrate data accordingly.

For brevity's sake, I'm not including any UI code, user prompts or error handling. You should ask (or at the very least inform) the user before doing any migration.

- (void)storesDidChange:(NSNotification *)notification
{
    NSDictionary *userInfo = notification.userInfo;
    NSPersistentStoreUbiquitousTransitionType transitionType = [[userInfo objectForKey:NSPersistentStoreUbiquitousTransitionTypeKey] integerValue];
    NSPersistentStore *persistentStore = [userInfo[NSAddedPersistentStoresKey] firstObject];
    id<NSCoding> ubiquityIdentityToken = [NSFileManager defaultManager].ubiquityIdentityToken;

    NSUserDefaults *defaults = [NSUserDefaults standardUserDefaults];
    if (transitionType != NSPersistentStoreUbiquitousTransitionTypeInitialImportCompleted) { // We only care of cases if the store was added or removed
        NSData *previousArchivedUbiquityIdentityToken = [defaults objectForKey:HPDefaultsUbiquityIdentityTokenKey];
        if (previousArchivedUbiquityIdentityToken) { // Was using ubiquity store
            if (!ubiquityIdentityToken) { // Changed to local account
                NSString *urlString = [defaults objectForKey:HPDefaultsPersistentStoreURLKey];
                NSURL *previousPersistentStoreURL = [NSURL URLWithString:urlString];
                [self importPersistentStoreAtURL:previousPersistentStoreURL
                 isLocal:NO
                 intoPersistentStore:persistentStore];
            }
        } else { // Was using local account
            if (ubiquityIdentityToken) { // Changed to ubiquity store
                NSString *urlString = [defaults objectForKey:HPDefaultsPersistentStoreURLKey];
                NSURL *previousPersistentStoreURL = [NSURL URLWithString:urlString];
                [self importPersistentStoreAtURL:previousPersistentStoreURL 
                 isLocal:YES 
                 intoPersistentStore:persistentStore];
            }
        }
    }
    if (ubiquityIdentityToken) {
        NSData *archivedUbiquityIdentityToken = [NSKeyedArchiver archivedDataWithRootObject:ubiquityIdentityToken];
        [defaults setObject:archivedUbiquityIdentityToken forKey:HPModelManagerUbiquityIdentityTokenKey];
    } else {
        [defaults removeObjectForKey:HPModelManagerUbiquityIdentityTokenKey];
    }
    NSString *urlString = persistentStore.URL.absoluteString;
    [defaults setObject:urlString forKey:HPDefaultsPersistentStoreURLKey];
    dispatch_async(dispatch_get_main_queue(), ^{
        // Update UI
    });
}

Then:

- (void)importPersistentStoreAtURL:(NSURL*)importPersistentStoreURL 
    isLocal:(BOOL)isLocal 
    intoPersistentStore:(NSPersistentStore*)persistentStore 
{
    if (!isLocal) { 
        // Create a copy because we can't add an ubiquity store as a local store without removing the ubiquitous metadata first,
        // and we don't want to modify the original ubiquity store.
        importPersistentStoreURL = [self copyPersistentStoreAtURL:importPersistentStoreURL];
    }
    if (!importPersistentStoreURL) return;

    // You might want to use a different concurrency type, depending on how you handle migration and the concurrency type of your current context
    NSManagedObjectContext *importContext = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSConfinementConcurrencyType];
    importContext.persistentStoreCoordinator = [[NSPersistentStoreCoordinator alloc] initWithManagedObjectModel:self.managedObjectModel];
    NSPersistentStore *importStore = [importContext.persistentStoreCoordinator
            addPersistentStoreWithType:NSSQLiteStoreType
            configuration:nil
            URL:importPersistentStoreURL
            options:@{NSPersistentStoreRemoveUbiquitousMetadataOption : @(YES)}
            error:nil];

    [self importContext:importContext intoContext:_managedObjectContext];
    if (!isLocal) {
        [[NSFileManager defaultManager] removeItemAtURL:importPersistentStoreURL error:nil];
    }
}

The data migration is performed in importContext:intoContext. This logic will depend of your model and duplicate and conflict policies.

I can't tell if this might have unwanted side effects. Obviously it might take quite a while depending on the size and data of the persistent store. If I find any problems I will edit the answer.