How to skip/mark incomplete entire test suite in PHPUnit?

Question

Description

I have a TestSuite which I need to mark as skipped (the entire test suite - not the specific test cases within the suite).

class AllTests
{    
    public static function suite()
    {
        // this does not work same as within TestCase:
        // throw new \PHPUnit_Framework_SkippedTestError("Out of order");

        $Suite = new \PHPUnit_Framework_TestSuite(__NAMESPACE__);
        $Suite->addTestSuite(translators\AllTests::cls());
        $Suite->addTestSuite(TlScopeTest::cls());
        $Suite->addTestSuite(TlNsTest::cls());
        $Suite->addTestSuite(TlElementTest::cls());
        $Suite->addTestSuite(TlItemTest::cls());
        $Suite->addTestSuite(LangCodeTest::cls());
        $Suite->addTestSuite(TlElemClassTagTest::cls());
        return $Suite;
    }
}

As you can see throwing the PHPUnit_Framework_SkippedTestError exception does not work. It is not caught by the PHPUnit, and is breaks the execution as any uncaught exception (which is understandable, as the suite() method is invoked while building tests hierarchy, before actually running the tests).

I've seen an exception class named PHPUnit_Framework_SkippedTestSuiteError, but have no clue how to take advantage of it. Any ideas?

Motivation

I have a TestSuite, which aggregates many test cases as well as other test suites. Almost every test in this fails, becouse of a change which I made in the core of my code.

The problem is that this package is not crutial, and is scheduled to be fixed later. Until then I have to run tests for every other package, but when I do the PHPUnit output becomes flooded with the errors coming from the package in question. This forces me to check every time if any of the failures is coming from any other package.

This, as you might suspect, is very susceptible to human error, i.e. I could miss a failure, which actually is important.

I could run only the test suite on which I am currently working, but I lose control of whether or not my changes in one package causes a failure in other package.

I do not want to comment out that test suite, because I'm afraid that I (or someone who will take over the code after me) could forget about it entirely.

Answer 1

Ok, so I'll put it together:

1. The AllTests class has to be refactored to extend PHPUnit_Framework_TestSuite.
2. This makes the class a fully valuable TestSuite and allows to implement a setUp method on the suite level.
3. The setUp method is called by the test runner (not by the builder), so it is safe to throw a SkippedTestError exception.
4. The corresponding method to do just that within a test suite is called markTestSuiteSkipped (notice the Suite in the method name).

The entire class would look like this:

class AllTests extends \PHPUnit_Framework_TestSuite
{    
    protected function setUp()
    {
        $this->markTestSuiteSkipped("zzz");
    }

    public static function suite()
    {
        $Suite = new self(__NAMESPACE__);
        $Suite->addTestSuite(translators\AllTests::cls());
        $Suite->addTestSuite(TlScopeTest::cls());
        $Suite->addTestSuite(TlNsTest::cls());
        $Suite->addTestSuite(TlElementTest::cls());
        $Suite->addTestSuite(TlItemTest::cls());
        $Suite->addTestSuite(LangCodeTest::cls());
        $Suite->addTestSuite(TlElemClassTagTest::cls());
        return $Suite;
    }
}

The output is a pretty block of S letters, which definetly indicate, that we skipped a lot of tests. This cannot escape our attention and yet allows our tests to pass.

Answer 2

You could mark test as skipped.