php/symfony/doctrine memory leak?

Question

I'm having problems with a batch insertion of objects into a database using symfony 1.4 and doctrine 1.2.

My model has a certain kind of object called "Sector", each of which has several objects of type "Cupo" (usually ranging from 50 up to 200000). These objects are pretty small; just a short identifier string and one or two integers. Whenever a group of Sectors are created by the user, I need to automatically add all these instances of "Cupo" to the database. In case anything goes wrong, I'm using a doctrine transaction to roll back everything. The problem is that I can only create around 2000 instances before php runs out of memory. It currently has a 128MB limit, which should be more than enough for handling objects that use less than 100 bytes. I've tried increasing the memory limit up to 512MB, but php still crashes and that doesn't solve the problem. Am I doing the batch insertion correctly or is there a better way?

Here's the error:

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 71 bytes) in /Users/yo/Sites/grifoo/lib/vendor/symfony/lib/log/sfVarLogger.class.php on line 170

And here's the code:

public function save($conn=null){

    $conn=$conn?$conn:Doctrine_Manager::connection();

    $conn->beginTransaction();


    try {
        $evento=$this->object;


        foreach($evento->getSectores() as $s){

            for($j=0;$j<$s->getCapacity();$j++){

                $cupo=new Cupo();
                $cupo->setActivo($s->getActivo());
                $cupo->setEventoId($s->getEventoId());
                $cupo->setNombre($j);
                $cupo->setSector($s);

                $cupo->save();

            }
        }

        $conn->commit();
        return;
    }
    catch (Exception $e) {
        $conn->rollback();
        throw $e;
    }

Once again, this code works fine for less than 1000 objects, but anything bigger than 1500 fails. Thanks for the help.

Answer 1

Tried doing

$cupo->save();
$cupo->free();
$cupo = null;

(But substituting my code) And I'm still getting memory overflows. Any other ideas, SO?

Update:

I created a new environment in my databases.yml, that looks like:

all:
  doctrine:
    class: sfDoctrineDatabase
    param:
      dsn: 'mysql:host=localhost;dbname=.......'
      username: .....
      password: .....
      profiler: false

The profiler: false entry disables doctrine's query logging, that normally keeps a copy of every query you make. It didn't stop the memory leakage, but I was able to get about twice as far through my data importing as I was without it.

Update 2

I added

Doctrine_Manager::connection()->setAttribute(Doctrine_Core::ATTR_AUTO_FREE_QUERY_OBJECTS, true ); 

before running my queries, and changed

$cupo = null;

to

unset($cupo);

And now my script has been churning away happily. I'm pretty sure it will finish without running out of RAM this time.

Update 3

Yup. That's the winning combo.

Answer 2

I have just did "daemonized" script with symfony 1.4 and setting the following stopped the memory hogging:

sfConfig::set('sf_debug', false);

Answer 3

For a symfony task, I also faced to this issue and done following things. It worked for me.

• Disable debug mode. Add following before db connection initialize

  sfConfig::set('sf_debug', false);
  

• Set auto query object free attribute for db connection

  $connection->setAttribute(Doctrine_Core::ATTR_AUTO_FREE_QUERY_OBJECTS, true );
  

• Free all object after use

  $object_name->free()
  

• Unset all arrays after use unset($array_name)

• Check all doctrine queries used on task. Free all queries after use. $q->free() (This is a good practice for any time of query using.)

That's all. Hope it may help someone.