Symfony2 / Doctrine2 throwing index error when flushing the entity manager

Question

Three entities are involved here: Deployment, DeploymentStep, and DeploymentStatusLog. I'll start by pasting the relevant definitions of those classes

src/My/Bundle/Entity/Deployment.php

<?php

namespace My\Bundle\Entity;

use Doctrine\ORM\Mapping as ORM;
use Doctrine\Common\Collections\ArrayCollection;
use Doctrine\ORM\PersistentCollection;

/**
 * @ORM\Table(name="deployment")
 * @ORM\Entity()
 */
class Deployment
{
  /**
   * Status Log Entries for this deployment
   *
   * @var \Doctrine\ORM\PersistentCollection
   *
   * @ORM\OneToMany(targetEntity="DeploymentStatusLog", mappedBy="deployment", cascade={"persist","remove"})
   * @ORM\OrderBy({"created_at"="DESC"})
   */
  protected $status_logs;

  /**
   * @var \Doctrine\ORM\PersistentCollection
   *
   * @ORM\OneToMany(targetEntity="DeploymentStep", mappedBy="deployment", cascade={"persist","remove"})
   * @ORM\OrderBy({"sequence" = "ASC"})
   */
  protected $steps;

  public function __construct()
  {
    $this->status_logs     = new ArrayCollection();
    $this->steps           = new ArrayCollection();
  }

  /**
   * Add status_logs
   *
   * @param DeploymentStatusLog $statusLogs
   */
  public function addDeploymentStatusLog(DeploymentStatusLog $statusLogs)
  {
      $this->status_logs[] = $statusLogs;
  }

  /**
   * Add steps
   *
   * @param DeploymentStep $steps
   */
  public function addDeploymentStep(DeploymentStep $steps)
  {
    $this->steps[] = $steps;
  }

  // ... 
}

src/My/Bundle/Entity/DeploymentStep.php

<?php

namespace My\Bundle\Entity;

use Doctrine\ORM\Mapping as ORM;

/**
 * @ORM\Table(name="deployment_step")
 * @ORM\Entity()
 */
class DeploymentStep
{
  /**
   * @var Deployment
   *
   * @ORM\ManyToOne(targetEntity="Deployment", cascade={"all"})
   * @ORM\JoinColumn(name="deployment_id", referencedColumnName="id")
   * @Gedmo\SortableGroup
   */
  private $deployment;

  /**
   * Set deployment
   *
   * @param Deployment $deployment
   */
  public function setDeployment(Deployment $deployment)
  {
    $this->deployment = $deployment;
  }

  // ...
}

src/My/Bundle/Entity/DeploymentStatusLog.php

<?php

namespace My\Bundle\Entity;

use Doctrine\ORM\Mapping as ORM;

/**
 * @ORM\Table(name="deployment_status_log")
 * @ORM\Entity()
 */
class DeploymentStatusLog
{
  /**
   * @var Deployment
   *
   * @ORM\ManyToOne(targetEntity="Deployment", cascade={"all"})
   * @ORM\JoinColumn(name="deployment_id", referencedColumnName="id", nullable=false)
   */
  protected $deployment;

  /**
   * Set deployment
   *
   * @param Deployment $deployment
   */
  public function setDeployment( Deployment $deployment)
  {
      $this->deployment = $deployment;
  }

  // ...
}

Now, the problem arises when I attempt to create brand new records for all three of these entities at once. In the controller:

$em = $this->getDoctrine()->getEntityManager();

$deployment = new Deployment();

$form = $this->createForm(new DeploymentType($em), $deployment);

if ($request->getMethod() == 'POST')
{
  $form->bindRequest($request);

  if ($form->isValid())
  {
    $codeStep = new DeploymentStep();
    $codeStep->setDeployment( $deployment );
    // Other setters on DeploymentStep

    $deploymentStatusLog = new DeploymentStatusLog();
    $deploymentStatusLog->setDeployment( $deployment );
    // Other setters on DeploymentStatusLog

    $deployment->addDeploymentStep( $codeStep );
    $deployment->addDeploymentStatusLog( $deploymentStatusLog );

    $em->persist( $deployment );
    $em->flush();
  }
}

What happens when the UnitOfWork processes, it throws a weird-looking exception complaining about an undefined index:

exception 'ErrorException' with message 'Notice: Undefined index: 000000001294f822000000006b6f9f2c in /project/vendor/doctrine/lib/Doctrine/ORM/UnitOfWork.php line 2252' in /project/vendor/symfony/src/Symfony/Component/HttpKernel/Debug/ErrorHandler.php:67

Now, if I persist/flush the Deployment entity first, and then persist/flush the associations, it succeeds.

So while I can do that to make this part of the application functional, it feels kinda wrong, since this process should be atomic and well, that's the whole point of transactional queries to begin with.

Any clues?

• Symfony 2.0.15
• Doctrine 2.1.7
• PHP 5.3.3
• MySQL 5.1.52
• Apache 2.2.15

EDIT

Full stack trace by request

 exception 'ErrorException' with message 'Notice: Undefined index: 000000004081f5f9000000005f1dbbfc in /project/vendor/doctrine/lib/Doctrine/ORM/UnitOfWork.php line 2252' in /project/vendor/symfony/src/Symfony/Component/HttpKernel/Debug/ErrorHandler.php:67
Stack trace:
#0 /project/vendor/doctrine/lib/Doctrine/ORM/UnitOfWork.php(2252): Symfony\Component\HttpKernel\Debug\ErrorHandler->handle(8, 'Undefined index...', '/mnt/hgfs/mount...', 2252, Array)
#1 /project/vendor/doctrine/lib/Doctrine/ORM/Query.php(321): Doctrine\ORM\UnitOfWork->getEntityIdentifier(Object(My\Bundle\Entity\Deployment))
#2 /project/vendor/doctrine/lib/Doctrine/ORM/Query.php(274): Doctrine\ORM\Query->processParameterValue(Object(My\Bundle\Entity\Deployment))
#3 /project/vendor/doctrine/lib/Doctrine/ORM/Query.php(243): Doctrine\ORM\Query->processParameterMappings(Array)
#4 /project/vendor/doctrine/lib/Doctrine/ORM/AbstractQuery.php(607): Doctrine\ORM\Query->_doExecute()
#5 /project/vendor/doctrine/lib/Doctrine/ORM/AbstractQuery.php(413): Doctrine\ORM\AbstractQuery->execute(Array, 1)
#6 /project/vendor/gedmo-doctrine-extensions/lib/Gedmo/Sortable/SortableListener.php(344): Doctrine\ORM\AbstractQuery->getResult()
#7 /project/vendor/gedmo-doctrine-extensions/lib/Gedmo/Sortable/SortableListener.php(133): Gedmo\Sortable\SortableListener->getMaxPosition(Object(Doctrine\ORM\EntityManager), Object(Doctrine\ORM\Mapping\ClassMetadata), Array, Object(My\Bundle\Entity\DeploymentStep))
#8 /project/vendor/gedmo-doctrine-extensions/lib/Gedmo/Sortable/SortableListener.php(100): Gedmo\Sortable\SortableListener->processInsert(Object(Doctrine\ORM\EntityManager), Array, Object(Doctrine\ORM\Mapping\ClassMetadata), Object(My\Bundle\Entity\DeploymentStep))
#9 /project/vendor/doctrine-common/lib/Doctrine/Common/EventManager.php(64): Gedmo\Sortable\SortableListener->onFlush(Object(Doctrine\ORM\Event\OnFlushEventArgs))
#10 /project/vendor/doctrine/lib/Doctrine/ORM/UnitOfWork.php(280): Doctrine\Common\EventManager->dispatchEvent('onFlush', Object(Doctrine\ORM\Event\OnFlushEventArgs))
#11 /project/vendor/doctrine/lib/Doctrine/ORM/EntityManager.php(334): Doctrine\ORM\UnitOfWork->commit()
#12 /project/src/My/Bundle/Controller/DeploymentController.php(214): Doctrine\ORM\EntityManager->flush()
#13 [internal function]: My\Bundle\Controller\DeploymentController->createAction(Object(My\Bundle\Entity\Release), Object(Symfony\Component\HttpFoundation\Request))
#14 /project/vendor/bundles/JMS/SecurityExtraBundle/Security/Authorization/Interception/MethodSecurityInterceptor.php(73): ReflectionMethod->invokeArgs(Object(My\Bundle\Controller\DeploymentController), Array)
#15 /project/app/cache/dev/classes.php(9391) : eval()'d code(1): JMS\SecurityExtraBundle\Security\Authorization\Interception\MethodSecurityInterceptor->invoke(Object(JMS\SecurityExtraBundle\Security\Authorization\Interception\MethodInvocation), Array)
#16 [internal function]: {closure}(Object(My\Bundle\Entity\Release), Object(Symfony\Component\HttpFoundation\Request))
#17 /project/app/cache/dev/classes.php(3925): call_user_func_array(Object(Closure), Array)
#18 /project/app/cache/dev/classes.php(3895): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1)
#19 /project/app/cache/dev/classes.php(4899): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#20 /project/app/bootstrap.php.cache(551): Symfony\Bundle\FrameworkBundle\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#21 /project/web/app_dev.php(18): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request))
#22 {main}

EDIT 2

Full code of create action, as requested

/**
 * @Route("/create/{id}", name="deployment_create_id")
 * @ParamConverter("release", class="MyBundle:Release")
 * @Method({"POST","GET"})
 * @Secure(roles="ROLE_DEPLOYMENT_PLANNER")
 * @Template()
 */
public function createAction( Release $release, Request $request )
{
  $em           = $this->getDoctrine()->getEntityManager();
  $sessionUser = $this->get('security.context')->getToken()->getUser();

  $deployment = new Deployment();
  $deployment->setRelease( $release );
  $deployment->setAuthor( $sessionUser );

  $form = $this->createForm(new DeploymentType($em), $deployment);

  if ($request->getMethod() == 'POST')
  {
    $form->bindRequest($request);

    if ($form->isValid())
    {
      $codeStep = new DeploymentStep();
      $codeStep->setDeployment( $deployment );
      $codeStep->setSequence( 0 );
      $codeStep->setTitle( "Update Code" );
      $codeStep->setDetails( "Update codebase per the plan's specifications" );
      $codeStep->setDeploymentStepType(
          $em->getRepository('MyBundle:DeploymentStepType')->findOneBy(
              array( 'name' => DeploymentStepType::TYPE_OTHER )
          )
      );

      $deploymentStatusLog = new DeploymentStatusLog();
      $deploymentStatusLog->setDeployment( $deployment );
      $deploymentStatusLog->setUser( $sessionUser );
      $deploymentStatusLog->setNotes( 'New Deployment Created' );
      $deploymentStatusLog->setDeploymentStatus(
          $em->getRepository('MyBundle:DeploymentStatus')->findOneBy(
              array( 'title' => DeploymentStatus::STATUS_NEW )
          )
      );

      $deployment->addDeploymentStep( $codeStep );
      $deployment->addDeploymentStatusLog( $deploymentStatusLog );

      try {
        $em->persist( $deployment );
           $em->persist( $codeStep );
           $em->persist( $deploymentStatusLog );
        $em->flush();

        return $this->redirectSuccess(
            'Deployment created.'
          , $release->getRouteName()
          , $release->getRouteParameters()
        );
      }
      catch ( \Exception $e )
      {
        $this->setFlashErrorMessage( 'Error saving deployment.' );
      }
    }
  }

  return array(
      'release' => $release
    , 'form'    => $form->createView()
  );
}

Answer 1

I had this same problem, in my case it was because I was doing an entity remove & flush in a postRemove LifeCycle Event. From what I could grok in the UnitOfWork logic, you cannot call flush in that event. Doing so will flush any actions still pending to be flushed from the first call to flush, which will already have been removed by the flush call in the event. After figuring that out I was able to find this section of the Doctrine manual that confirmed my suspicion:

http://docs.doctrine-project.org/en/latest/reference/events.html#postupdate-postremove-postpersist

You are correct to assume that flushing each step individually is a bad approach, but if you don't make use of LifeCycle events then I'm not sure what could be causing your particular problem. I was able to debug the issue by error logging the object id variable ($oid) in the executeDeletions function of UnitOfWork.php. I noticed the same id was being deleted repeatedly, which once unset from $this->entityIdentifiers, will fail on subsequent deletes.

My solution was to simply catalog each ID in the postRemove event, and then actually remove the entities in a postFlush event, which is not a lifecycle event and therefore can make subsequent persist operations:

http://docs.doctrine-project.org/en/latest/reference/events.html#lifecycle-events

I'm sure you have since moved on, but in case anyone else runs into this problem...

Answer 2

This is not a "solution" but I hope it provides more troubleshooting information to others.

I had this same exact error when following Doctrine's recommendation for batch processing mass inserts. FYI, this is from a controller (not a lifecycle event like the other answer mentions).

Doctrine's Recommended Method

$batchSize = 20;
for ($i = 1; $i <= 10000; ++$i) {
    $user = new CmsUser;
    $user->setStatus('user');
    $user->setUsername('user' . $i);
    $user->setName('Mr.Smith-' . $i);
    $em->persist($user);
    if (($i % $batchSize) === 0) {
        $em->flush();
        $em->clear(); // Detaches all objects from Doctrine!
    }
}
$em->flush(); //Persist objects that did not make up an entire batch
$em->clear();

When I used something similar to the recommended code above, it would fail with the same errors after about 2000 inserts.

Change Order of Persist

The order I persisted the 10,000 entities made no difference. And it made no difference if I persisted, flushed and cleared every single loop (not ideal, but I tried it).

Remove Clear

If I just commented out the $em->clear() within the $batchSize check and made it only clear after the loop finished, then it timed out:

Fatal error: Maximum execution time of 30 seconds exceeded in /var/www/core/cms/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php on line 541

So I put a set_time_limit(3600) on the script to prevent time out, and it no longer gave the error, but it ran out of memory :P

This would suggest that the problem occurs when $em->clear() is executed within the loop. This is consistent with other questions. Unfortunately, without $em->clear(), you run out of memory quickly.

Disable Event Listeners

The other answer mentions that Event Listeners may be causing this, so I disabled them like it suggested:

foreach ($em->getEventManager()->getListeners() as $event => $listeners) {
    foreach ($listeners as $listener) {
        $em->getEventManager()->removeEventListener($event, $listener);
    }
}

But that didn't work either... although it seems like this could be the issue, somehow, and that this doesn't actually successfully disable them.

Validate Schema

I also validated my schema:

php app/console doctrine:schema:validate

And no errors are reported.

[Mapping]  OK - The mapping files are correct.
[Database] OK - The database schema is in sync with the mapping files.