Core data executeFetchRequest consumes huge amounts of memory

Question

I am inserting cca 100 000 records in core data database. Database contains 3 entities. Player, Club, PlayerClub Entities are in relations: Player<-->>PlayerClub<<-->Club While inserting in PlayerClub I have noticed a lot of memory consumption and loss of speed after about 50 000 records have been inserted. Records are never updated cause PlayerClub has only unique values. For test reasons I have emptied Player and Club tables and run the program again. There was no speed drop but memory consumption was huge again. This is the code:

    NSManagedObjectContext *context = [[NSManagedObjectContext alloc] init];
    [context setPersistentStoreCoordinator:ap.persistentStoreCoordinator];
    [context setUndoManager:nil];

    NSEntityDescription *entity = [NSEntityDescription entityForName:@"PlayerClub"
                                              inManagedObjectContext:context];

    NSFetchRequest *request = [[NSFetchRequest alloc] init];
    [request setEntity:entity];

    NSEntityDescription *entityKlubovi = [NSEntityDescription entityForName:@"Club" inManagedObjectContext:context];
    NSFetchRequest *requestKlubovi = [[NSFetchRequest alloc] init];
    [requestKlubovi setEntity:entityKlubovi];
    [requestKlubovi setFetchLimit:1];

    NSEntityDescription *entityIgraci = [NSEntityDescription entityForName:@"Player" inManagedObjectContext:context];
    NSFetchRequest *requestIgraci = [[NSFetchRequest alloc] init];
    [requestIgraci setFetchLimit:1];
    [requestIgraci setEntity:entityIgraci];

    for (int i=0; i<[parsedKlubIgraci count]; i++) {
        @autoreleasepool {
            KlubIgrac *klubIgrac = [[KlubIgrac alloc] initWithEntity:entity insertIntoManagedObjectContext:context];

            klubIgrac.iD = p.id;

            //... add some othe properties

            variables = [NSDictionary dictionaryWithObject:p.igracID forKey:@"ID"];
            localPredicate = [predicateIgraci predicateWithSubstitutionVariables:variables];
            [requestIgraci setPredicate:localPredicate];
            klubIgrac.igrac = [self DohvatiIgracZaIgracId:p.igracID cntx:context request:requestIgraci predicate:localPredicate];

            variables = [NSDictionary dictionaryWithObject:p.klubID forKey:@"ID"];
            localPredicate = [predicateKlubovi predicateWithSubstitutionVariables:variables];
            [requestKlubovi setPredicate:localPredicate];
            klubIgrac.klub = [self DohvatiKlubZaKlubId:p.klubID cntx:context request:requestKlubovi predicate:localPredicate];
        }

    }
+(Club *)DohvatiKlubZaKlubId:(int)klubid cntx:(NSManagedObjectContext *)context    

    request:(NSFetchRequest *)request predicate:(NSPredicate *)predicate
{
    @autoreleasepool {
        NSError *error;
        NSArray *arTmp;
        arTmp = [context executeFetchRequest:request error:&error];
        Club *klub;
        if(arTmp != nil && [arTmp count]){
            return [arTmp objectAtIndex:0];
        }
    }

}

DohvatiIgracZaIgracId method is basically the same method as DohvatiKlubZaKlubId so I dinnt post it. This code is called about 100 000 times. The memory consumption before is about 150 MB. After it finishes its 650 MB. So it consumes 500 MB, without saving the context and without fetching anything from the database cause the tables are empty. If I comment

arTmp = [context executeFetchRequest:request error:&error];

line in DohvatiIgracZaIgracId and DohvatiKlubZaKlubId methods the memory consumption falls to 200 MB. So the diference is 400MB for a line of code that does nothing. This cant be. Anyone has some ideas. After a while my app consumes more than 2.5 GB.

The mesuments were done on simulator cause I need to build a database that I will preload later, so the speed is of importance :)... Thx in advance

Answer 1

without saving the context

This is the part of the question where experienced Core Data developers say "oh holy crap". That's your biggest problem right there. Save changes at regular intervals-- every 50 entries, or every 100, but whatever you do, don't wait until you're finished. You're forcing Core Data to keep all of those objects in memory as unsaved changes. This is the biggest reason you're having memory problems.

Some other things you should consider:

• Don't fetch your objects one at a time. Fetches are relatively expensive. If you run through 100k instances and fetch each of them one at a time, your code will be spending almost all of its time executing fetches. Fetch in batches of 50-100 (you can tune the number to get the best balance of speed vs. memory use). Process one batch, then save changes at the end of the batch.

• When you're done with a fetched object, tell the managed object context that you're done. Do this by calling refreshObject:mergeChanges: with NO as the second argument. That tells the context it can free up any internal memory it's using for the object. This loses any unsaved changes on the objects, but if you haven't made any changes then there's nothing to lose.

• Consider getting rid of the PlayerClub entity completely. Core Data supports many-to-many relationships. This kind of entity is almost never useful. You're using Core Data, not SQL, so don't design your entity types as if you were.