Core Data Import - Not releasing memory

Question

My question is about Core Data and memory not being released. I am doing a sync process importing data from a WebService which returns a json. I load in, in memory, the data to import, loop through and create NSManagedObjects. The imported data needs to create objects that have relationships to other objects, in total there are around 11.000. But to isolate the problem I am right now only creating the items of the first and second level, leaving the relationship out, those are 9043 objects.

I started checking the amount of memory used, because the app was crashing at the end of the process (with the full data set). The first memory check is after loading in memory the json, so that the measurement really takes only in consideration the creation, and insert of the objects into Core Data. What I use to check the memory used is this code (source)

-(void) get_free_memory {

    struct task_basic_info info;
    mach_msg_type_number_t size = sizeof(info);
    kern_return_t kerr = task_info(mach_task_self(), 
                                   TASK_BASIC_INFO,
                                   (task_info_t)&info,
                                   &size);
    if( kerr == KERN_SUCCESS ) {
        NSLog(@"Memory in use (in bytes): %f",(float)(info.resident_size/1024.0)/1024.0 );
    } else {
        NSLog(@"Error with task_info(): %s", mach_error_string(kerr));
    }
}

My setup:

• 1 Persistent Store Coordinator
• 1 Main ManagedObjectContext (MMC) (NSMainQueueConcurrencyType used to read (only reading) the data in the app)
• 1 Background ManagedObjectContext (BMC) (NSPrivateQueueConcurrencyType, undoManager is set to nil, used to import the data)

The BMC is independent to the MMC, so BMC is no child context of MMC. And they do not share any parent context. I don't need BMC to notify changes to MMC. So BMC only needs to create/update/delete the data.

Plaform:

• iPad 2 and 3
• iOS, I have tested to set the deployment target to 5.1 and 6.1. There is no difference
• XCode 4.6.2
• ARC

Problem: Importing the data, the used memory doesn't stop to increase and iOS doesn't seem to be able to drain the memory even after the end of the process. Which, in case the data sample increases, leads to Memory Warnings and after the closing of the app.

Research:

1. Apple documentation

   • Efficiently importing Data
   • Reducing Memory Overhead

2. Good recap of the points to have in mind when importing data to Core Data (Stackoverflow)

3. Tests done and analysis of the memory release. He seems to have the same problem as I, and he sent an Apple Bug report with no response yet from Apple. (Source)

4. Importing and displaying large data sets (Source)

5. Indicates the best way to import large amount of data. Although he mentions:

   "I can import millions of records in a stable 3MB of memory without calling -reset."

   This makes me think this might be somehow possible? (Source)

Tests:

Data Sample: creating a total of 9043 objects.

• Turned off the creation of relationships, as the documentation says they are "expensive"
• No fetching is being done

Code:

- (void)processItems {
    [self.context performBlock:^{
        for (int i=0; i < [self.downloadedRecords count];) {
            @autoreleasepool
            {
                [self get_free_memory]; // prints current memory used
                for (NSUInteger j = 0; j < batchSize && i < [self.downloadedRecords count]; j++, i++)
                {
                    NSDictionary *record = [self.downloadedRecords objectAtIndex:i];

                    Item *item=[self createItem];
                    objectsCount++;

                    // fills in the item object with data from the record, no relationship creation is happening
                    [self updateItem:item WithRecord:record];

                    // creates the subitems, fills them in with data from record, relationship creation is turned off
                    [self processSubitemsWithItem:item AndRecord:record]; 
                }
                // Context save is done before draining the autoreleasepool, as specified in research 5)
                [self.context save:nil];

                // Faulting all the created items
                for (NSManagedObject *object in [self.context registeredObjects]) {
                    [self.context refreshObject:object mergeChanges:NO];
                }
                // Double tap the previous action by reseting the context
                [self.context reset];
            }
        }
    }];
    [self check_memory];// performs a repeated selector to [self get_free_memory] to view the memory after the sync 
}

Measurment:

It goes from 16.97 MB to 30 MB, after the sync it goes down to 28 MB. Repeating the get_memory call each 5 seconds maintains the memory at 28 MB.

Other tests without any luck:

• recreating the persistent store as indicated in research 2) has no effect
• tested to let the thread wait a bit to see if memory restores, example 4)
• setting context to nil after the whole process
• Doing the whole process without saving context at any point (loosing therefor the info). That actually gave as result maintaing less amount of memory, leaving it at 20 MB. But it still doesn't decrease and... I need the info stored :)

Maybe I am missing something but I have really tested a lot, and after following the guidelines I would expect to see the memory decreasing again. I have run Allocations instruments to check the heap growth, and this seems to be fine too. Also no memory Leaks.

I am running out of ideas to test/adjust... I would really appreciate if anyone could help me with ideas of what else I could test, or maybe pointing to what I am doing wrong. Or it is just like that, how it is supposed to work... which I doubt...

Thanks for any help.

EDIT

I have used instruments to profile the memory usage with the Activity Monitor template and the result shown in "Real Memory Usage" is the same as the one that gets printed in the console with the get_free_memory and the memory still never seems to get released.

Answer 1

Ok this is quite embarrassing... Zombies were enabled on the Scheme, on the Arguments they were turned off but on Diagnostics "Enable Zombie Objects" was checked...

Turning this off maintains the memory stable.

Thanks for the ones that read trough the question and tried to solve it!

Answer 2

It seems to me, the key take away of your favorite source ("3MB, millions of records") is the batching that is mentioned -- beside disabling the undo manager which is also recommended by Apple and very important).

I think the important thing here is that this batching has to apply to the @autoreleasepool as well.

It's insufficient to drain the autorelease pool every 1000 iterations. You need to actually save the MOC, then drain the pool.

In your code, try putting a second @autoreleasepool into the second for loop. Then adjust your batch size to fine-tune.

I have made tests with more than 500.000 records on an original iPad 1. The size of the JSON string alone was close to 40MB. Still, it all works without crashes, and some tuning even leads to acceptable speed. In my tests, I could claim up to app. 70MB of memory on an original iPad.