Synchronize IOS core data with web service?

Question

Here is my problem:

• I want to use core data - speed and connectivity issues to build my IOS app. The data stored in core data is coming from a SQLServer database which I can access through a yet-to-be-defined web service.
• Any changes to the data stored in core data needs to be synchronized with the SQLServer via a web service. In addition, I need to buffer changes that don't get synchronized because of connectivity issues.
• I also need to update core data with any changes that have occured on the server. This could happen on a schedule set in user preferences.

Solutions I've Explored:

• Using NSIncrementalStore class (new in IOS 5). I'm very confused on what this does exactly but it sounds promising. From what I can tell, you subclass NSIncrementalStore which allows you to intercept the regular core data API calls. I could then pass on the the information to core data as well as sync it with the external database via a web service. I could be completely wrong. But assuming I'm right, how would I sync deltas if the connection to the internet is down?
• AFIncrementalStore - This is a subclass off of NSIncrementalStore using AFNetworking to do the web services piece.
• RestKit - I'm a little concerned on how active this API is and it seems to be going through a transition to block functionality. Has anyone used this extensively?

I'm leaning towards AFIncrementalStore since this is using (what seems to be) a more standard approach. The problem is, I could be completely off on what NSIncrementalStore really is.

A link to some sample code or tutorial would be great!

Answer 1

My solution to this was to store two copies of the data set in a CoreData database. One represents the last-known server state and is immutable. The other is edited by the user.

When it is time to sync changes, the app creates a diff between the edited and immutable copies of the data. The app sends the diff to a web service which applies the diff to its own copy of the data. It replies with a full copy of the data set, which the app overwrites onto both of its copies of the data.

The advantages are:

• If there is no network connectivity, no changes are lost: the diff is calculated each time the data set needs to be sent, and the immutable copy is only changed on a successful sync.
• Only the minimum amount of information that needs to be sent is transmitted.
• Multiple people can edit the same data at the same time without using locking strategies with a minimum opportunity for data loss via overwrites.

The disadvantages are:

• Writing the diffing code is complex.
• Writing the merging service is complex.
• Unless you are a metaprogramming guru, you'll find that your diff/merge code is brittle and has to change whenever you change your object model.

Here are some of the considerations I had when coming up with the strategy:

• If you allow changes to be made offline, checkin/checkout locking won't work (how can you establish a lock with no connection?).
• What happens if two people edit the same data at the same time?
• What happens if one person edits data on one iOS device when connectionless, switches it off, edits on another device and then turns the original device back on?
• Multithreading with CoreData is an entire problem class in itself.

The closest thing I've heard of to out-of-the-box support to do anything remotely like this is the new iCloud/CoreData syncing system in iOS6, which automatically transmits entities from a CoreData database to iCloud when they change. However, that means you have to use iCloud.

EDIT: This is very late, I know, but here's a class that is capable of producing a diff between two NSManagedObject instances.

// SZManagedObjectDiff.h
@interface SZManagedObjectDiff

- (NSDictionary *)diffNewObject:(NSManagedObject *)newObject withOldObject:(NSManagedObject *)oldObject

@end

// SZManagedObjectDiff.m
#import "SZManagedObjectDiff.h"

@implementation SZManagedObjectDiff

- (NSDictionary *)diffNewObject:(NSManagedObject *)newObject withOldObject:(NSManagedObject *)oldObject {

    NSDictionary *attributeDiff = [self diffAttributesOfNewObject:newObject withOldObject:oldObject];

    NSDictionary *relationshipsDiff = [self diffRelationshipsOfNewObject:newObject withOldObject:oldObject];

    NSMutableDictionary *diff = [NSMutableDictionary dictionary];

    if (attributeDiff.count > 0) {
        diff[@"attributes"] = attributeDiff;
    }

    if (relationshipsDiff.count > 0) {
        diff[@"relationships"] = relationshipsDiff;
    }

    if (diff.count > 0) {
        diff[@"entityName"] = newObject ? newObject.entity.name : oldObject.entity.name;

        NSString *idAttributeName = newObject ? newObject.entity.userInfo[@"id"] : oldObject.entity.userInfo[@"id"];

        if (idAttributeName) {
            id itemId = newObject ? [newObject valueForKey:idAttributeName] : [oldObject valueForKey:idAttributeName];

            if (itemId) {
                diff[idAttributeName] = itemId;
            }
        }
    }

    return diff;
}

- (NSDictionary *)diffRelationshipsOfNewObject:(NSManagedObject *)newObject withOldObject:(NSManagedObject *)oldObject {

    NSMutableDictionary *diff = [NSMutableDictionary dictionary];

    NSDictionary *relationships = newObject == nil ? [[oldObject entity] relationshipsByName] : [[newObject entity] relationshipsByName];

    for (NSString *name in relationships) {

        NSRelationshipDescription *relationship = relationships[name];

        if (relationship.deleteRule != NSCascadeDeleteRule) continue;

        SEL selector = NSSelectorFromString(name);

        id newValue = nil;
        id oldValue = nil;

        if (newObject != nil && [newObject respondsToSelector:selector]) newValue = [newObject performSelector:selector];
        if (oldObject != nil && [oldObject respondsToSelector:selector]) oldValue = [oldObject performSelector:selector];

        if (relationship.isToMany) {

            NSArray *changes = [self diffNewSet:newValue withOldSet:oldValue];

            if (changes.count > 0) {
                diff[name] = changes;
            }

        } else {

            NSDictionary *relationshipDiff = [self diffNewObject:newValue withOldObject:oldValue];

            if (relationshipDiff.count > 0) {
                diff[name] = relationshipDiff;
            }
        }
    }

    return diff;
}

- (NSDictionary *)diffAttributesOfNewObject:(NSManagedObject *)newObject withOldObject:(NSManagedObject *)oldObject {

    NSMutableDictionary *diff = [NSMutableDictionary dictionary];

    NSArray *attributeNames = newObject == nil ? [[[oldObject entity] attributesByName] allKeys] : [[[newObject entity] attributesByName] allKeys];

    for (NSString *name in attributeNames) {

        SEL selector = NSSelectorFromString(name);

        id newValue = nil;
        id oldValue = nil;

        if (newObject != nil && [newObject respondsToSelector:selector]) newValue = [newObject performSelector:selector];
        if (oldObject != nil && [oldObject respondsToSelector:selector]) oldValue = [oldObject performSelector:selector];

        newValue = newValue ? newValue : [NSNull null];
        oldValue = oldValue ? oldValue : [NSNull null];

        if (![newValue isEqual:oldValue]) {
            diff[name] = @{ @"new": newValue, @"old": oldValue };
        }
    }

    return diff;
}

- (NSArray *)diffNewSet:(NSSet *)newSet withOldSet:(NSSet *)oldSet {

    NSMutableArray *changes = [NSMutableArray array];

    // Find all items that have been newly created or updated.
    for (NSManagedObject *newItem in newSet) {

        NSString *idAttributeName = newItem.entity.userInfo[@"id"];

        NSAssert(idAttributeName, @"Entities must have an id property set in their user info.");

        id newItemId = [newItem valueForKey:idAttributeName];

        NSManagedObject *oldItem = nil;

        for (NSManagedObject *setItem in oldSet) {
            id setItemId = [setItem valueForKey:idAttributeName];

            if ([setItemId isEqual:newItemId]) {
                oldItem = setItem;
                break;
            }
        }

        NSDictionary *diff = [self diffNewObject:newItem withOldObject:oldItem];

        if (diff.count > 0) {
            [changes addObject:diff];
        }
    }

    // Find all items that have been deleted.
    for (NSManagedObject *oldItem in oldSet) {

        NSString *idAttributeName = oldItem.entity.userInfo[@"id"];

        NSAssert(idAttributeName, @"Entities must have an id property set in their user info.");

        id oldItemId = [oldItem valueForKey:idAttributeName];

        NSManagedObject *newItem = nil;

        for (NSManagedObject *setItem in newSet) {
            id setItemId = [setItem valueForKey:idAttributeName];

            if ([setItemId isEqual:oldItemId]) {
                newItem = setItem;
                break;
            }
        }

        if (!newItem) {
            NSDictionary *diff = [self diffNewObject:newItem withOldObject:oldItem];

            if (diff.count > 0) {
                [changes addObject:diff];
            }
        }
    }

    return changes;
}

@end

There's more information about what it does, how it does it and its limitations/assumptions here:

http://simianzombie.com/?p=2379