Should all unit tests be successful?

Question

Let's say you make an application that attempts to transliterate stuff from alphabet A into alphabet B, as closely as possible.

Because language B is very complex, this is not always successful. But you do get an approximate transliteration.

How would you build unit tests in this case, considering that you expect 20-30% to fail?

Answer 1

It must always be the goal that your unit test is successful. The way you use it, you can not differentiate between serious errors in your software and the errors in the translations you seem to expect.

I would suggest to separate both:

1. Use unit tests only to make sure that your software works the way you expect it (even if this includes some mistakes in the translations)
2. Produce a test dataset containing the translations produced by your software. You can check this test dataset either manually or using a statistical software package like R to get an idea of the quality of your translations.

Answer 2

Unit test should be deterministic. Failing test should indicate software failure, not something that "works as intended". For your case, prepare data in a way you can be sure of results and test for them, whether conversion succeeds or fails (testing for failure is always an option, given you expect failure - important is that you're always in control of when your test passes/fails).

Answer 3

A unit test that is expected to fail is not a unit test. You need to change the definition of success by using an evaluation function that acts as a filter and decides if it is "close enough" and determines pass/fail. As your translator gets better, you can raise the bar in the filter.