Advantages of formal unit testing over just writing a lot of examples? [closed]

Question

R has three well-developed unit testing packages, RUnit, svUnit, and testthat. Base R has examples built into its package functionality, which will return an error if they do not parse properly.

The opinion of those whom I trust is that unit testing is better than writing lots of examples, but I can't quite put my finger on any specific functionality in unit tests that can't be replicated in examples.

What features of using a unit testing framework in R make it superior to the ad hoc equivalent using package examples?

For those not from the R world, note that examples for every function in a package are run each time the package is built, and the programmer is made to suffer for any warnings or errors.

Answer 1

With a unit test framework you can test all kinds of things that you may not want to expose as examples to the end user:

• Does the function throw the correct error with incorrect input? A unit test framework will catch the error, thus passing the test.
• Unit testing should aim to get really high coverage of the code. You may want to test all of the combinations of all of the input arguments.

Another advantage of unit test frameworks is speed:

• You can selectively run tests in a given file, for example. In contrast, depending on package output means you need to build and check the entire package.
• This leads to much quicker identification of errors and a quicker development cycle.

My typical packages would contain dozens, if not hundreds of tests, but only a few examples that really demonstrate what the package is about.

In summary, I use tests to test, and examples to educate and help.

Answer 2

It of course depends which kind of examples you have, but most probably the examples are not showing how not to use the code. Unit tests can also test behaviour in negative cases.

Unit tests probably test smaller parts and therefore simpler functionalities. Tests that use frameworks can be typically run automatically after each code change. If you rely on someone running the examples all the time, sooner or later they are forgotten in some critical moment.

Answer 3

One of the obvious difference between tests and examples, is that tests can be a part of the development cycle. In test-driven development TDD each new feature begins with writing a test.

I will no detail here the TDD concept ( I hate generality and I think the topic is well developed in the literature) But If you always finish a session of coding by creating a failing test for the feature you want to implement next, it will be easier for you to pick up where you left off when you want to continue your development.