How to handle large doctrine Collections and Associations

Question

I'm working on a project where I have to collect a lot of data from the database. I'm using Symfony2 (symfony bf1281aebdc842a39ec0eb7438e1ea3fca9b9705) and Doctrine2 (doctrine 3b3186ee98392802a44118cd421a3530119aa7eaand ) as working base.

The Problem I encounter is, that I have to fetch about 15.000 articles. After that, I need to iterate through all of them an fetch more data on base of the article id (there are direct and indirect associations with i.e. (inherited) media or prices, etc...). It's ok for about 50-100 records, but if I want to use more records it takes a lot of ram an time to fetch everything from the database.

Is there a way to iterate through the data without using up all of the remaining ram? Is there a way to tell doctrine to stop using references?

Thank you in advance for any help!

Answer 1

save another line by using:

$iterableResult = $doctrine->getManager()->createQuery("SELECT c FROM ENTITY c")->iterate();

while ((list($obj) = $iterableResult->next()) !== false) {
    // do something with $obj
    $em->detach($obj);
}

Answer 2

You can look into using iterative (step-by-step) hydration in Doctrine:

$em = $this->getDoctrine()->getEntityManager();
$q  = $em->createQuery("<DQL to select the objects I want>");
$iterableResult = $q->iterate();
while (($row = $iterableResult->next()) !== false) {
    // do stuff with the data in the row, $row[0] is always the object
    $em->detach($row[0]); // detach from Doctrine, so that it can be GC'd immediately
}

Details from this article, under Mass object processing