PHPUnit not continuing with tests after fatal error when using --process-isolation

Question

I have a PHPUnit test suite that is currently causing a fatal error due to a class definition that's not being found. This, ultimately, is a failure of the testing code itself and a failure by the developer to vindicate the test itself before committing code.

However, things like this do happen from time to time, and it would be wonderful if, when a fatal error occurs (regardless of who's ultimately responsible), the test simply be marked as a failure, and the remainder of the test suite still be executed.

I have read about the --process-isolation switch and as far as I can tell, it should take care of this. Since each test runs in a separate process, if the child dies due to a fatal error, the parent can still continue to run. In fact, this is stated explicitly in this answer to a similar question: https://stackoverflow.com/a/5340151/84762 which shows the exact type of output I would like to see myself.

However, I seem to get the exact same output regardless of whether or not I use the --process-isolation flag:

WITHOUT process isolation

[kogi@phagocyte ~]$ /usr/bin/phpunit --colors --verbose --coverage-html "target/coverage" ~/app/zend/tests/application/

PHP Fatal error:  Class 'Rmd_Database_OldObject' not found in /home/kogi/app/zend/private/models/translate/Poll.php on line 9
PHP Stack trace:
PHP   1. {main}() /usr/bin/phpunit:0
PHP   2. PHPUnit_TextUI_Command::main() /usr/bin/phpunit:46
PHP   3. PHPUnit_TextUI_Command->run() /usr/share/pear/PHPUnit/TextUI/Command.php:130
PHP   4. PHPUnit_Runner_BaseTestRunner->getTest() /usr/share/pear/PHPUnit/TextUI/Command.php:150
PHP   5. PHPUnit_Framework_TestSuite->addTestFiles() /usr/share/pear/PHPUnit/Runner/BaseTestRunner.php:96
PHP   6. PHPUnit_Framework_TestSuite->addTestFile() /usr/share/pear/PHPUnit/Framework/TestSuite.php:419
PHP   7. PHPUnit_Util_Fileloader::checkAndLoad() /usr/share/pear/PHPUnit/Framework/TestSuite.php:358
PHP   8. PHPUnit_Util_Fileloader::load() /usr/share/pear/PHPUnit/Util/Fileloader.php:79
PHP   9. include_once() /usr/share/pear/PHPUnit/Util/Fileloader.php:95
PHP  10. require_once() /home/kogi/app/zend/tests/application/translate/PollTest.php:11

Fatal error: Class 'Rmd_Database_OldObject' not found in /home/kogi/app/zend/private/models/translate/Poll.php on line 9

Call Stack:
    0.0003      91584   1. {main}() /usr/bin/phpunit:0
    0.0076     612672   2. PHPUnit_TextUI_Command::main() /usr/bin/phpunit:46
    0.0076     613744   3. PHPUnit_TextUI_Command->run() /usr/share/pear/PHPUnit/TextUI/Command.php:130
    0.0246    1249464   4. PHPUnit_Runner_BaseTestRunner->getTest() /usr/share/pear/PHPUnit/TextUI/Command.php:150
    0.0706    1626680   5. PHPUnit_Framework_TestSuite->addTestFiles() /usr/share/pear/PHPUnit/Runner/BaseTestRunner.php:96
    0.1691    8053584   6. PHPUnit_Framework_TestSuite->addTestFile() /usr/share/pear/PHPUnit/Framework/TestSuite.php:419
    0.1693    8057320   7. PHPUnit_Util_Fileloader::checkAndLoad() /usr/share/pear/PHPUnit/Framework/TestSuite.php:358
    0.1694    8057664   8. PHPUnit_Util_Fileloader::load() /usr/share/pear/PHPUnit/Util/Fileloader.php:79
    0.1711    8240600   9. include_once('/home/kogi/app/zend/tests/application/translate/PollTest.php') /usr/share/pear/PHPUnit/Util/Fileloader.php:95
    0.1805    9187768  10. require_once('/home/kogi/app/zend/private/models/translate/Poll.php') /home/kogi/app/zend/tests/application/translate/PollTest.php:11

WITH process isolation

[kogi@phagocyte ~]$ /usr/bin/phpunit --colors --verbose --coverage-html "target/coverage" --process-isolation ~/app/zend/tests/application/
PHP Fatal error:  Class 'Rmd_Database_OldObject' not found in /home/kogi/app/zend/private/models/translate/Poll.php on line 9
PHP Stack trace:
PHP   1. {main}() /usr/bin/phpunit:0
PHP   2. PHPUnit_TextUI_Command::main() /usr/bin/phpunit:46
PHP   3. PHPUnit_TextUI_Command->run() /usr/share/pear/PHPUnit/TextUI/Command.php:130
PHP   4. PHPUnit_Runner_BaseTestRunner->getTest() /usr/share/pear/PHPUnit/TextUI/Command.php:150
PHP   5. PHPUnit_Framework_TestSuite->addTestFiles() /usr/share/pear/PHPUnit/Runner/BaseTestRunner.php:96
PHP   6. PHPUnit_Framework_TestSuite->addTestFile() /usr/share/pear/PHPUnit/Framework/TestSuite.php:419
PHP   7. PHPUnit_Util_Fileloader::checkAndLoad() /usr/share/pear/PHPUnit/Framework/TestSuite.php:358
PHP   8. PHPUnit_Util_Fileloader::load() /usr/share/pear/PHPUnit/Util/Fileloader.php:79
PHP   9. include_once() /usr/share/pear/PHPUnit/Util/Fileloader.php:95
PHP  10. require_once() /home/kogi/app/zend/tests/application/translate/PollTest.php:11

Fatal error: Class 'Rmd_Database_OldObject' not found in /home/kogi/app/zend/private/models/translate/Poll.php on line 9

Call Stack:
    0.0003      91752   1. {main}() /usr/bin/phpunit:0
    0.0076     612824   2. PHPUnit_TextUI_Command::main() /usr/bin/phpunit:46
    0.0076     613896   3. PHPUnit_TextUI_Command->run() /usr/share/pear/PHPUnit/TextUI/Command.php:130
    0.0246    1250360   4. PHPUnit_Runner_BaseTestRunner->getTest() /usr/share/pear/PHPUnit/TextUI/Command.php:150
    0.0708    1627528   5. PHPUnit_Framework_TestSuite->addTestFiles() /usr/share/pear/PHPUnit/Runner/BaseTestRunner.php:96
    0.1688    8054296   6. PHPUnit_Framework_TestSuite->addTestFile() /usr/share/pear/PHPUnit/Framework/TestSuite.php:419
    0.1690    8057992   7. PHPUnit_Util_Fileloader::checkAndLoad() /usr/share/pear/PHPUnit/Framework/TestSuite.php:358
    0.1691    8058336   8. PHPUnit_Util_Fileloader::load() /usr/share/pear/PHPUnit/Util/Fileloader.php:79
    0.1707    8241296   9. include_once('/home/kogi/app/zend/tests/application/translate/PollTest.php') /usr/share/pear/PHPUnit/Util/Fileloader.php:95
    0.1801    9188464  10. require_once('/home/kogi/app/zend/private/models/translate/Poll.php') /home/kogi/app/zend/tests/application/translate/PollTest.php:11

For those of us that can't effectively diff in our heads, the two outputs are literally identical (other than the execution time and memory usage which is negligibly different).

In both cases, the fatal error kills the entire test suite. In this particular case, this happens in the 3rd test and the remaining 150 tests (in several other files/suites) are never executed.

What am I doing wrong here? Is there some other way to survive a fatal error (marking the test as failed) in one test and still execute remaining tests?

---

EDIT

I am using PHPUnit 3.6.10

EDIT

Comments on the answer to this question have inspired a new ticket on PHPUnit's GitHub page: https://github.com/sebastianbergmann/phpunit/issues/545

Answer 1

PHPUnit loads each test file that will be run before running any tests. This causes PHP to parse these files and execute their top-level code. If any class is loaded that, for example, extends a class that doesn't exist, you'll get a fatal error.

I don't see any way around this without enhancing PHPUnit to parse the files without executing their code during the scanning process.