How to run some but not all tests in a Perl test suite in parallel?

Question

I've got a Perl-based test suite with 10,000+ tests that I would like to make run faster. I've tested using the -j flag to prove, and I have found that most-but-not-all of my tests are ready to run in parallel.

While I can work on making the remaining tests to be "parallel friendly", I expect there always be some tests which are not. What's a good way to manage this? I would like for it to be easy to run the whole set of tests efficiently, and make it easy to mark tests as "not-parallel-ready" if I need to.

Here are some options I see:

1. prove could be patched to support some tests as not-parallel-ready
2. Jenkins is being used to manage the test suite runs. I could split off the non-parallel tests into their own run. In other words, give up and use two test runs.
3. Perhaps there is a way to merge two TAP result streams together that I have yet to recover.

I'm not too concerned with how I will manage the list of exceptions. Either I can keep a list in a file as part of the test harness infrastructure, or I could put something in each test header that would mark it as such, and our test harness could determine the list of exceptions dynamically.

( The test suite is partially based on Test::Class, and I'll also be looking at Test::Class::Load to speed it up as well. )

Answer 1

I found a solution. It's in the documentation for aggregate_tests() for TAP::Harness. It includes a code sample for how I could write my own harness for this purpose:

...This is useful, for example, in the case where some tests should run in parallel but others are unsuitable for parallel execution.

my $formatter   = TAP::Formatter::Console->new;
my $ser_harness = TAP::Harness->new( { formatter => $formatter } );
my $par_harness = TAP::Harness->new(
    {   formatter => $formatter,
        jobs      => 9
    }
);
my $aggregator = TAP::Parser::Aggregator->new;

$aggregator->start();
$ser_harness->aggregate_tests( $aggregator, @ser_tests );
$par_harness->aggregate_tests( $aggregator, @par_tests );
$aggregator->stop();
$formatter->summary($aggregator);

From there it looks like I could:

• Sub-class App::Prove and override _runtests(), which is where the new functionality above could be merged in.
• Fork prove so that it calls My::App::Prove instead of App::Prove.

Now that I better understand how the pieces fit together I can see how I might create a patch for prove that would add an option like --exclude-from-parallel FILE, which would allow you to specify a file, which contains a list of test files to be excluded from parallel testing.

UPDATE 2012-08-16: I have a patch for prove now, and have submitted it for review. You can view and comment on the Pull Request. No summary is produced after the run output. It's not clear why.

Answer 2

I've now found the best solution so far to this problem. In turns out that prove has had undocumented support for marking some tests to be run in sequence instead of parallel since 2008. It's backed by a rather fancy "rules" system in TAP::Parser::Scheduler that allows for complex specifications of ordering arrangements for parallel and sequential test runs.

Here's the basic current recipe for prove:

 # All tests are allowed to run in parallel, except those starting with "p"
 --rules='seq=t/p*.t' --rules='par=**'

I have a new pull request that adds documentation for this feature, and have started a discussion about possibly offering a simpler syntax for basic exceptions as well. See the pull request for details.