NSFetchedResultsController attempting to insert nil object

Question

Edit 7:

Here's my save method. It's pretty boilerplate. The DEBUG_LOG() macros are only executed if it's a debug build.

- (void)saveManagedObjectContext:(NSManagedObjectContext *)moc
{
    if ([moc hasChanges]) {
        DEBUG_LOG(@"Saving managed object context %@", moc);
        NSError *error;
        BOOL success = [moc save:&error];
        if (!success || error) {
            DEBUG_LOG(@"ERROR: Couldn't save to managed object context %@: %@",
                  moc, error.localizedDescription);
        }
        DEBUG_LOG(@"Finished saving managed object context %@", moc);
    } else {
        DEBUG_LOG(@"Managed object context %@ had no changes", moc);
    }
}

Edit 6:

iOS 8 is here and this problem is back. Lucky me. Previously I had narrowed the problem down to using estimatedRowHeight on table views (btw, I never fully fixed the problem. I just stopped using estimatedRowHeight). Now I'm seeing this problem again under different circumstances. I tracked it down to a commit from a few days ago when I made my nav/tab bars translucent. This included disabling 'Adjust Scroll View Insets' in storyboard and checking the boxes to have my views display under top bars and bottom bars. There's a series of steps I need to do to make it happen, but I can reproduce it every time with my storyboard configured that way. If I revert that commit it no longer happens.

While I say, "it no longer happens," what I really think is that this is just making it less likely to happen. This bug is an absolute b****. My gut reaction now is that this is an iOS bug. I just don't know what I can do to turn this into a bug report. It's madness.

Edit 5:

If you want to read the entirety of my misery, please continue all the way through this post. If you're running into this problem and you just want some help, here's something to look into.

My last edit noted that when I used a basic table view cell, everything worked fine. My next course of action was going to be to try from scratch building a new custom cell piece by piece and seeing where it messed up. For the hell of it, I re-enabled my old custom cell code and it worked just fine. Uhhh? Oh wait, I still have estimatedHeightForRowAtIndexPath commented out. When I removed those comments and enabled estimatedHeightForRowAtIndexPath, it got crappy again. Interesting.

I looked up that method in the API doc, and it mentioned something about a constant called UITableViewAutomaticDimension. The value I was estimating was really just one of the common cell heights, so it wouldn't hurt to switch to that constant. After switching to that constant it's working properly. No weird exceptions/graphical glitches to report.

Original post

I have a pretty standard iPhone app that fetches data from a web service in the background and displays data in a table view. The background updating work has its own managed object context configured for NSPrivateQueueConcurrencyType. My table view's fetched results controller has its own managed object context configured for NSMainQueueConcurrencyType. When the background context parses new data it passes that data to the main context via mergeChangesFromContextDidSaveNotification. Sometimes during the merge, my app hits an exception here...

Thread 1, Queue : com.apple.main-thread
#0  0x3ac1b6a0 in objc_exception_throw ()
#1  0x308575ac in -[__NSArrayM insertObject:atIndex:] ()
#2  0x33354306 in __46-[UITableView _updateWithItems:updateSupport:]_block_invoke687 ()
#3  0x330d88d2 in +[UIView(UIViewAnimationWithBlocks)     _setupAnimationWithDuration:delay:view:options:factory:animations:start:animationStateGenerator:completion:] ()
#4  0x330ef7e4 in +[UIView(UIViewAnimationWithBlocks) animateWithDuration:delay:options:animations:completion:] ()
#5  0x3329e908 in -[UITableView _updateWithItems:updateSupport:] ()
#6  0x332766c6 in -[UITableView _endCellAnimationsWithContext:] ()
#7  0x0005ae72 in -[ICLocalShowsTableViewController controllerDidChangeContent:] at ICLocalShowsTableViewController.m:475
#8  0x3069976c in -[NSFetchedResultsController(PrivateMethods) _managedObjectContextDidChange:] ()
#9  0x308dfe78 in __CFNOTIFICATIONCENTER_IS_CALLING_OUT_TO_AN_OBSERVER__ ()
#10 0x30853b80 in _CFXNotificationPost ()
#11 0x3123a054 in -[NSNotificationCenter postNotificationName:object:userInfo:] ()
#12 0x306987a2 in -[NSManagedObjectContext(_NSInternalNotificationHandling) _postObjectsDidChangeNotificationWithUserInfo:] ()
#13 0x306f952a in -[NSManagedObjectContext _mergeChangesFromDidSaveDictionary:usingObjectIDs:] ()
#14 0x306f9734 in -[NSManagedObjectContext mergeChangesFromContextDidSaveNotification:] ()
#15 0x0006b5be in __65-[ICManagedObjectContexts backgroundManagedObjectContextDidSave:]_block_invoke at ICManagedObjectContexts.m:133
#16 0x306f9854 in developerSubmittedBlockToNSManagedObjectContextPerform ()
#17 0x3b1000ee in _dispatch_client_callout ()
#18 0x3b1029a8 in _dispatch_main_queue_callback_4CF ()
#19 0x308e85b8 in __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ ()
#20 0x308e6e84 in __CFRunLoopRun ()
#21 0x30851540 in CFRunLoopRunSpecific ()
#22 0x30851322 in CFRunLoopRunInMode ()
#23 0x355812ea in GSEventRunModal ()
#24 0x331081e4 in UIApplicationMain ()
#25 0x000554f4 in main at main.m:16

Here's the exception I see...

CoreData: error: Serious application error.  An exception was caught from the delegate of NSFetchedResultsController during a call to -controllerDidChangeContent:.  *** -[__NSArrayM insertObject:atIndex:]: object cannot be nil with userInfo (null)

My app is actually hitting the exception in controllerDidChangeContent, at my call to endUpdates. I'm basically seeing the same thing as this (NSFetchedResultsController attempting to insert nil object?), but I've got more info and case a that's reproducible. All of my merge events are inserts. During the merge, there doesn't seem to be any pending inserts, deletes, or updates on the background context. I was initially using performBlockAndWait all over the place until I learned about the difference between performBlock and performBlockAndWait from the WWDC video. I switched to performBlock, and that made it a little bit better. Initially I approached this as a threading issue, diverged into the possibility of it being a weird memory problem caused by not fully understanding blocks, and now I'm back to it being a race condition. It seems like there's just one piece that I'm missing. There are two ways it doesn't happen...

(1) Register for the context will save notification, nil out the FRC delegate when I get it, and set the delegate back after the merge. This isn't far from not using an FRC at all, so this really isn't an option for a workaround.

(2) Do things that block the main thread long enough, so the race condition doesn't happen. For example, when I add a lot of debug log messages to my table view delegate, that slows it down enough for it not to happen.

Here are what I believe to be the important pieces of code (I've shortened certain spots to shrink this already large post).

After various points during scrolling, the view controller will request more data by calling a function that has this in it...

AFJSONRequestOperation *operation =
    [AFJSONRequestOperation JSONRequestOperationWithRequest:request
        success:^(NSURLRequest *request, NSHTTPURLResponse *response, id JSON) {
            // Parsing happens on MOC background queue
            [backgroundMOC performBlock:^ {
                [self parseJSON:JSON];

                // Handle everything else on the main thread
                [mainMOC performBlock:^ {
                    if (completion) {
                        // Remove activitiy indicators and such from the main thread
                    }
                }];
            }];
        }

        failure:^(NSURLRequest *request, NSHTTPURLResponse *response, NSError *error, id JSON) {
            [[NSOperationQueue mainQueue] performBlock:^ {
                if (completion) {
                    // Remove activitiy indicators and such from the main thread
                }
                // Show an alert view saying that the request failed
            }];
        }
     ];

[operation setCacheResponseBlock:^NSCachedURLResponse *(NSURLConnection *connection, NSCachedURLResponse *cachedResponse) {
    return nil;
}];

[_operationQueue addOperation:operation];

For the most part, parseJSON doesn't really have anything interesting in it...

- (void)parseJSON:(NSDictionary *)json
{       
    NSError *error;
    NSArray *idExistsResults;
    NSNumber *eventId;
    NSFetchRequest *idExistsFetchRequest;
    LastFMEvent *event;
    NSManagedObjectModel *model = backgroundMOC.persistentStoreCoordinator.managedObjectModel;
    for (NSDictionary *jsonEvent in jsonEvents) {
        eventId = [NSNumber numberWithInt:[jsonEvent[@"id"] intValue]];
        idExistsFetchRequest = [model fetchRequestFromTemplateWithName:kGetEventByIDFetchRequest substitutionVariables:@{@"eventID" : eventId}];
        idExistsResults  = [backgroundMOC executeFetchRequest:idExistsFetchRequest error:&error];
        // Here I check for errors - omitted that part

        if ([idExistsResults count] == 0) {
            // Add a new event
            event = [NSEntityDescription insertNewObjectForEntityForName:[LastFMEvent entityName] inManagedObjectContext:backgroundMOC];
            [event populateWithJSON:jsonEvent];
        } else if ([idExistsResults count] == 1) {
            // Get here if I knew about the event already, so I update a few fields
        }
    }
    [self.mocManager saveManagedObjectContext:backgroundMOC];
}

The implementation for save and merge are where it might get interesting. Save expects to be called from within the appropriate performBlock already, so it doesn't do anything with the performBlock.

- (void)saveManagedObjectContext:(NSManagedObjectContext *)moc
{
    if ([moc hasChanges]) {
        NSError *error;
        BOOL success = [moc save:&error];
        if (!success || error) {
            NSLog(@"ERROR: Couldn't save to managed object context %@: %@",
                  moc, error.localizedDescription);
        }
    }
}

Upon saving, the merge notification gets triggered. I'm only merging from background to main, so I pretty much just want to know if I can inline the merge call or if I need to do it inside of performBlock.

- (void)backgroundManagedObjectContextDidSave:(NSNotification *)notification
{
    if (![NSThread isMainThread]) {
        [mainMOC performBlock:^ {
            [self.mainMOC mergeChangesFromContextDidSaveNotification:notification];
        }];
    } else {
        [mainMOC mergeChangesFromContextDidSaveNotification:notification];
    }
}

My fetched results controller delegate methods are pretty boiler plate stuff...

- (void)controller:(NSFetchedResultsController *)controller
   didChangeObject:(id)anObject
       atIndexPath:(NSIndexPath *)indexPath
     forChangeType:(NSFetchedResultsChangeType)type
      newIndexPath:(NSIndexPath *)newIndexPath
{
    UITableView *tableView = self.tableView;

    switch (type) {

        case NSFetchedResultsChangeInsert:
            [tableView insertRowsAtIndexPaths:@[newIndexPath]
                             withRowAnimation:UITableViewRowAnimationAutomatic];
            break;
        case NSFetchedResultsChangeDelete:
            [tableView deleteRowsAtIndexPaths:@[indexPath]
                             withRowAnimation:UITableViewRowAnimationAutomatic];
            break;
        case NSFetchedResultsChangeUpdate:
            [self configureCell:(ICLocalShowsTableViewCell *)[tableView cellForRowAtIndexPath:indexPath]
                          atIndexPath:indexPath];
            break;
        case NSFetchedResultsChangeMove:
            [tableView deleteRowsAtIndexPaths:@[indexPath]
                             withRowAnimation:UITableViewRowAnimationAutomatic];
            [tableView insertRowsAtIndexPaths:@[newIndexPath]
                             withRowAnimation:UITableViewRowAnimationAutomatic];
            break;
    }
}

- (void)controller:(NSFetchedResultsController *)controller
  didChangeSection:(id )sectionInfo
           atIndex:(NSUInteger)sectionIndex
     forChangeType:(NSFetchedResultsChangeType)type
{
    switch(type) {

        case NSFetchedResultsChangeInsert:
            [self.tableView insertSections:[NSIndexSet indexSetWithIndex:sectionIndex]
                          withRowAnimation:UITableViewRowAnimationAutomatic];
            break;

        case NSFetchedResultsChangeDelete:
            [self.tableView deleteSections:[NSIndexSet indexSetWithIndex:sectionIndex]
                          withRowAnimation:UITableViewRowAnimationAutomatic];
            break;
    }
}

- (void)controllerWillChangeContent:(NSFetchedResultsController *)controller
{
    [self.tableView beginUpdates];
}

- (void)controllerDidChangeContent:(NSFetchedResultsController *)controller
{
    [self.tableView endUpdates];
}

One other piece of code that might be of interest. I'm using autolayout for my table view cells, and the new estimatedHeightForRowAtIndexPath API for dynamic cell height. What this means is that during the call to [self.tableView endUpdates], the last step actually reaches down into some managed objects, whereas the other calls for number of sections/rows only need to know counts from the FRC.

- (CGFloat)tableView:(UITableView *)tableView heightForRowAtIndexPath:(NSIndexPath *)indexPath
{
    NSAssert([NSThread isMainThread], @"");
    LastFMEvent *event = [self.fetchedResultsController objectAtIndexPath:indexPath];

    if (!_offscreenLayoutCell) {
        _offscreenLayoutCell = [self.tableView dequeueReusableCellWithIdentifier:kLocalShowsCellIdentifier];
    }

    [_offscreenLayoutCell configureWithLastFMEvent:event];
    [_offscreenLayoutCell setNeedsLayout];
    [_offscreenLayoutCell layoutIfNeeded];

    CGSize cellSize = [_offscreenLayoutCell.contentView systemLayoutSizeFittingSize:UILayoutFittingCompressedSize];
    return cellSize.height;
}

Been stuck on this for almost a week now. Learned a ton in the process, but geez I'm ready to move on. Any suggestions would be greatly appreciated.

Edit

I put together a pretty big debug log to try to tell the story of what's going on with the udpates. I'm seeing something really strange. I'm updating the table with 50 rows at a time, so I'll only include the interesting part of my debug output. Every time a cell gets configured I'm printing out what the title was for the cell that I just dequeued as well as what the new title will be. When I hit the last cell in the table view, I make a query to the web service for more data. This output is related to the final update before I hit the exception...

// Lots of output was here that I omitted

configure cell at sect 5 row 18 WAS Suphala NOW Keller Williams
configure cell at sect 5 row 19 WAS Advocate Of Wordz NOW Gates
configure cell at sect 5 row 20 WAS Emanuel and the Fear NOW Beats Antique
configure cell at sect 5 row 21 WAS The Julie Ruin NOW Ashrae Fax

// At this point I hit the end of the table and query for more data - for some reason row 18 gets configured again. Possibly no big deal.

configure cell at sect 5 row 18 WAS Keller Williams NOW Keller Williams
configure cell at sect 5 row 22 WAS Old Wounds NOW Kurt Vile

JSON size 100479
Starting JSON parsing
page 3 of 15. total events 709. events per page 50. current low idx 100 next trigger idx 149

// Parsing data finished, saving background context

Saving managed object context <NSManagedObjectContext: 0x17e912f0>
Background context will save
Finished saving managed object context <NSManagedObjectContext: 0x17e912f0>
Merging background context into main context
JSON parsing finished

** controllerWillChangeContent called **
** BEGIN UPDATES triggered **

inserting SECTION 6
inserting SECTION 7
inserting SECTION 8
inserting ROW sect 5 row 17
inserting ROW sect 5 row 22
inserting ROW sect 5 row 25
inserting ROW sect 5 row 26
inserting ROW sect 5 row 27
inserting ROW sect 5 row 28
inserting ROW sect 5 row 29

// A bunch more rows added here that I omitted

** controllerDidChangeContent called **

// This configure cell happens before the endUpdates call has completed

configure cell at sect 5 row 18 WAS Conflict NOW Conflict

In the final update it's attempting to insert at s5 r17, but I already had a cell at that row. It also attempts to insert at s5 r22, but I also already had a cell at that row. Lastly it inserts a row at s5 r25, which actually is a new row. It seems to me as though considering r17 and r22 as inserts is leaving a gap in the table. Shouldn't the previous cells at those indexes have events to be moved to r23 and r24?

My fetched results controller is using a sort descriptor that sorts by date and start time. Maybe the existing events that were at r17 and r22 aren't getting move events because there weren't any changes related to their NSManagedObjects. Essentially, they are required to move because of my sort descriptor for events earlier than them and not because their data changed.

Edit 2:

Looks like those inserts do just trigger the existing cells to shift down :(

Edit 3:

Things I tried today...

1. Made AFNetworking success block waits for the merge to complete before it returns
2. Made cellForRowAtIndexPath return a stale cell (essentially dequeue it and return it right away) if the fetched results controller is in the middle of beginUpdates/endUpdates. Thinking that extra random cellForRowAtIndexPath that gets called during the update may have been doing weird things.
3. Removing the background context altogether. This is interesting. If I do all of the UI updates AND JSON parsing on the main context, it still happens.

Edit 4:

Now it's getting interesting.

I tried removing random components in my table view such as the refresh control. Also tried getting rid of my use of estimatedHeightForRowAtIndexPath, which meant just supplying a static row height instead of using autolayout to determine dynamic row height. Both of those turned up nothing. I also tried getting rid of my custom cell entirely, and just using a basic table view cell.

That worked.

I tried a basic table view cell with subtitle.

That worked.

I tried a basic table view cell with subtitle and image.

That worked.

The top of my stack trace being near all of those animation related items is starting to make more sense. It's looking like this is auto-layout related.

Answer 1

From an Apple Technical Support Engineer:

To protect the integrity of the datastore, Core Data catches some exceptions that happen during its operations. Sometimes this means that if Core Data calls your code through a delegate method, Core Data may end up catching exceptions your code threw.

Multi-threading errors are the most common cause of mysterious Core Data issues.

In this case, Core Data caught an exception through your controllerDidChangeContent: method, caused by trying to use insertObject:atIndex.

The most likely fix is to ensure that all your NSManagedObject code is encapsulated inside performBlock: or performBlockAndWait: calls.

In iOS 8 and OSX Yosemite, Core Data gains the ability to detect and report violations of its concurrency model. It works by throwing an exception whenever your app accesses a managed object context or managed object from the wrong dispatch queue. You enable the assertions by passing -com.apple.CoreData.ConcurrencyDebug 1 to your app on the command line via Xcodeʼs Scheme Editor.

Ole Begemann has a great writeup of the new feature.