PHPUnit - Automatically retry failed tests X times?

Question

I have a complex set of PHPUnit tests, some of which involve connecting to servers around the world that, for whatever reason, timeout sometimes.

Rather than having the test fail when the server times out, I'd like to simply retry that test one or more times before actually marking it as failed.

Now, I understand that this may not be the best way to handle my situation at hand. One better solution would be to fix the servers. But, this is out of my control right now.

So, what I'd really like, is a way to tell PHPUnit to re-test each failing testcase X times, and only mark it as failed if it failed every time.

Any ideas?

Edit: Many of you have responded with helpful suggestions that I not do this. I understand, thank you. However, specifically what I am trying to do is create a test suite that tests the operation of the full system, including the remote servers. I understand the concept of testing certain parts of my code with "mock" responses from the outside...but I also sleep better at night if part of my tests test the "full stack".

Answer 1

As PHPUnit doesn't support this behavior out of the box, you'll need to code the loop yourself. Instead of doing it in every test that requires it, create a custom test case base class (if you haven't already) that extends PHPUnit_Framework_TestCase and provides the feature.

You can either get fancy and override testBare() to check for an annotation such as @retry 5, loop that number of times, calling parent::testBare(), and swallow all exceptions (or a subset) except the last.

public function runBare() {
    // I'll leave this part to you. PHPUnit supplies methods for parsing annotations.
    $retryCount = $this->getNumberOfRetries();
    for ($i = 0; $i < $retryCount; $i++) {
        try {
            parent::runBare();
            return;
        }
        catch (Exception $e) {
            // last one thrown below
        }
    }
    if ($e) {
        throw $e;
    }
}

Or you can create a similar helper method that takes the number of retries and a closure/callable as parameters and call it from each test that needs it.

public function retryTest($count, $test) {
    // just like above without checking the annotation
    ...
        $test();
    ...
}

public function testLogin() {
    $this->retryTest(5, function() {
        $service = new LoginService();
        ...
    });
}