How to plan additional tests run by external sub?

Question

We have written parsers for different scientific data formats in Perl. Recently I added a test suite with a parser_*.t file for every format and subformat.

Of course, the API of the parsers is exactly the same only the data read from the example files that are used for testing parsing differs. To simplify the test files I wrote a sub that gets passed the parser object and a hash structure representing the expected data. It looks like

my $parser = new MyApp::Parser($file);
test_nested_objects = ($parser, {
     property1 => "value",
     property2 => 123,
     subobject_accessor => {
         property3 => "foobar",
     }
}

The sub test_nested_objects walks through the hash and runs tests for all properties defined in the hash, e.g. if subobject_accessor can be called, returns an object and that object can be called property3.

I checked how many tests are run by the whole *.t file and added tests => 123 to all *.t files. Now, I added some checks to the generic function and all plans are wrong.

How to make my plan aware of the subtests? I'd like to achieve the following:

• number of tests given before running them for view of progess
• total number increased automatically → no changing of numbers by hand when editing the sub
• individual tests in sub visible when running prove (hiding tests in sub and returning just 0 or 1 isn't acceptable because I really need to know what is wrong with the parsed data)

I hope you can understand. Sorry for the long story but I thought people would probably not understand without some background knowledge.

Answer 1

The latest trend in test planning is simply not to do it. At the end of your test file, you declare that you reached the end:

 done_testing();

Many uses of a test plan exist only to ensure that the test script completed. If it saw the right number of tests, the script must have made it all the way to the end. That's not really true, but it's a convenient and workable fiction.

If you have to do some calculations to determine the number of tests, such as measuring the size of a data structure, do that then call plan later:

 use Test::More;  # don't declare the plan yet

 my $test_count = ...;

 plan tests => $test_count; 

Instead of making big plans, however, you can divide your tests into subtests. Each subtest only has to know how many tests it runs, and the script only has to know the number of subtests:

 subtest some_label => sub {
      plan tests => $n;
      ...
      };

Each of your calls to a testing function would be one test, and the function would use a subtest to do whatever it wanted to do without exposing too much to its caller.