Is there a Hierarchical Test Runner for PHPUnit?

Question

I've seen how Hierarchical Context Runner works in JUnit, and it's pretty awesome.

It allows you to arrange multiple setups before groups of methods in a single test class. This is great when you are testing multiple scenarios; it feels more like doing BDD. Hierarchical Runner Explanation

Would be nice to have something like this in PHPUnit, but I just can't achieve this.

I've tried using @before annotations over custom methods, hoping to prescribe the order. Also, I've tried to declare inner classes, but then I discovered that is not allowed in PHP 5. I've also tried many other things without success.

Is it possible to achieve this using PHPUnit?

Answer 1

You can't do exactly what the JUnit Heirarchical Context Runner does because, as you have discovered, you can't nest classes in PHP. Heirarchical Context Runner depends on nested classes. You can get very close to the same thing, however. Ultimately, with some thought to how you name your test methods, you can produce cleaner code that's easy to navigate and understand, with less risk of accidentally introducing global state or hidden dependencies than if you could use nested classes.

Important Caveat

Before we dive in, please note that you generally do not want to share fixtures or other state between tests. The whole point of unit testing is to test individual units of code, which is hard to do when you also create links between those units by having persistent (or worse, actually variable) data across tests. As explained in the PHPUnit docs,

There are few good reasons to share fixtures between tests, but in most cases the need to share a fixture between tests stems from an unresolved design problem. [emphasis added]

A good example of a fixture that makes sense to share across several tests is a database connection: you log into the database once and reuse the database connection instead of creating a new connection for each test. This makes your tests run faster.

Use a Bootstrap File

If you have code that should run once before all of your tests, across all of your test classes, use a bootstrap file. For example, if your code depends on an autoloader or a constant like the path containing a particular file, or if you just need to run a series of include or require statements to load some functions, use a bootstrap file.

You can do this with the --bootstrap command line option, like this:

phpunit --bootstrap src/autoload.php tests

You can also specify a bootstrap file in an XML configuration file, like so:

<phpunit bootstrap="src/autoload.php">
  <!-- other configuration stuff here -->
</phpunit>

Use Setup Methods

You can also specify a setup method to run before running any other tests in a particular class. This is where you would put all of the code that should run before any tests in the class. It will only run once, however, so you can't use it to run between tests.

For example, you could do this to populate data for one or more scenarios before running any tests:

<?php
class NameValidatorTest extends PHPUnit_Framework_TestCase
{
    protected static $nameForEnglishScenario;
    protected static $nameForFrenchScenario;

    /**
     * Runs before any tests and sets up data for the test
     */
    public static function setUpBeforeClass()
    {
        self::$nameForEnglishScenario = 'John Smith';
        self::$nameForFrenchScenario = 'Séverin Lemaître';
    }

    public function testEnglishName()
    {
        $this->assertRegExp('/^[a-z -]+$/i', self::$nameForEnglishScenario);
    }

    public function testFrenchName()
    {
        $this->assertRegExp('/^[a-zàâçéèêëîïôûùüÿñæœ -]+$/i', self::$nameForFrenchScenario);
    }
}

(Pay no attention to the actual logic in this sample; the tests here are lame and aren't actually testing a class. The focus is on the setup.)

Consider Using Multiple Test Methods Per Target Method within Your Test Class

The typical way to test multiple scenarios is to create multiple methods with names reflecting their conditions. For example, if I'm testing a name validator class, I might do something like this:

<?php
class NameValidatorTest extends PHPUnit_Framework_TestCase
{
    public function testIsValid_Invalid_English_Actually_French()
    {
        $validator = new NameValidator();
        $validator->setName('Séverin Lemaître');
        $validator->setLocale('en');
        $this->assertFalse($validator->isValid());
    }

    public function testIsValid_Invalid_French_Gibberish()
    {
        $validator = new NameValidator();
        $validator->setName('Séverin J*E08RW)8WER Lemaître');
        $validator->setLocale('fr');
        $this->assertFalse($validator->isValid());
    }

    public function testIsValid_Valid_English()
    {
        $validator = new NameValidator();
        $validator->setName('John Smith');
        $validator->setLocale('en');
        $this->assertTrue($validator->isValid());
    }

    public function testIsValid_Valid_French()
    {
        $validator = new NameValidator();
        $validator->setName('Séverin Lemaître');
        $validator->setLocale('fr');
        $this->assertTrue($validator->isValid());
    }
}

This has the advantage of consolidating all of your tests for a class in one place and, if you name them intelligently, making it easy to navigate, even with lots of test methods.

Consider Using Data Provider Methods

You can also use a data provider method. From the manual:

A test method can accept arbitrary arguments. These arguments are to be provided by a data provider method (provider() in Example 2.5). The data provider method to be used is specified using the @dataProvider annotation.

See the section called “Data Providers” for more details.

You can use data providers to run the same test code more than once, using different data for each run to test different scenarios.

Consider Using Dependencies

You can also force tests within a test class to run in a specific order by specifying dependencies between them. You do this using @depends in a docblock. An example from the documentation:

<?php
class MultipleDependenciesTest extends PHPUnit_Framework_TestCase
{
    public function testProducerFirst()
    {
        $this->assertTrue(true);
        return 'first';
    }

    public function testProducerSecond()
    {
        $this->assertTrue(true);
        return 'second';
    }

    /**
     * @depends testProducerFirst
     * @depends testProducerSecond
     */
    public function testConsumer()
    {
        $this->assertEquals(
            array('first', 'second'),
            func_get_args()
        );
    }
}

In this example, both testProducerFirst and testProducerSecond are guaranteed to run before testConsumer. Note, however, that testConsumer will receive as arguments the results of testProducerFirst and testProducerSecond, and it will not run at all if one of those tests fails.

Consider Using Multiple Test Classes per Target Class

If you want to run lots of tests on very different scenarios, you might consider creating more than one test class for a given target class. This is generally not your best bet, however. It means creating and maintaining more classes (and thus more files, if you're putting only one class in a file, as you should), and it makes it harder to see and understand all of your test code at once. This is only really appropriate for instances in which your target class is used in very different manners from one test to another.

If you're writing SOLID code and using design patterns properly, however, your code shouldn't be capable to running under such different conditions that this makes sense. So, this is somewhat a matter of opinion, but this is probably never the correct way to write your tests.

Consider Using Test Suites to Run Tests in a Particular Order

You can also tell PHPUnit to run a "test suite." This allows you to group tests in a logical manner (e.g., all database tests or all tests for classes with i18n logic). When you compose your test suite using an XML configuration file, you can tell PHPUnit explicitly to run your tests in a particular order. As explained in the docs,

If phpunit.xml or phpunit.xml.dist (in that order) exist in the current working directory and --configuration is not used, the configuration will be automatically read from that file.

The order in which tests are executed can be made explicit:

Example 5.2: Composing a Test Suite Using XML Configuration

   <phpunit bootstrap="src/autoload.php">
     <testsuites>
       <testsuite name="money">
         <file>tests/IntlFormatterTest.php</file>
         <file>tests/MoneyTest.php</file>
         <file>tests/CurrencyTest.php</file>
       </testsuite>
     </testsuites>
   </phpunit>

The downside of this is, again, you're introducing a form of global state. What if, in your real application, you use the Money class before running some critical functionality from the IntlFormatter class?

Putting It All Together

Your best bet is to use a setUpBeforeClass() method to do setup on a per-test-class basis. Then, use multiple test methods per target method to test out your various scenarios.

There are a number of other ways to force tests to run in a particular order, which I have set out above. But they all introduce some form of global state, clutter, or both. Any time you make one test only run after another completes, you risk introducing dependencies without realizing it. You're not really unit testing if your tests depend on each other. At some level, you're doing integrating testing, instead.

As a rule, you are better off doing true unit tests. Test each public method of a target class as though nothing else existed. When you can do that and make your tests pass for all conceivable scenarios, then you have reliable code.