Black Box Unit Testing

Question

In my last project, we had Unit Testing with almost 100% cc, and as a result we almost didn’t have any bugs.

However, since Unit Testing must be White Box (you have to mock inner functions to get the result you want, so your tests need to know about the inner structure of your code) any time we changed the implementation of a function, we had to change the tests as well.

Note that we didn't change the logic of the functions, just the implementation.

It was very time-consuming and it felt as if we are working the wrong way. Since we used all proper OOP guidelines (specifically Encapsulation), every time we changed the implementation we didn't have to change the rest of our code, but had to change the unit tests.

It felt as if we are serving the tests, instead of them serving us.

To prevent this, some of us argued that unit tests should be Black Box Testing.

That would be possible if we create one big mock of our entire Domain and create a stub for every function in every class in one place, and use it in every unit test.

Of course that if a specific test needs specific inner function to be called (Like making sure we write to the DB), we can override our stub.

So, every time we change the implementation of a function (like adding or replacing a call to a help function) we will only need to change our main big mock. Even if we do need to change some unit tests, it will still be much less than before.

Others argue that unit tests must be White Box, since not only do you want to make sure your app writes to the DB in a specific place, you want to make sure your app does not write to the DB anywhere else unless you specifically expect it to. While this is a valid point, I don't think it worth the time of writing White Box tests instead of Black Box tests.

So in conclusion, two questions:

1. What do you think about the concept of Black Box Unit Testing?

2. What do you think about the way we want to implement that concept? Do you have better ideas?

Answer 1

You need different types of tests.

• Unit-tests which should be white-box testing, as you did

• Integration tests (or system tests) which test the ability to use the actual implementations of your system and its communication with external layers (external systems, database, etc.) which should be black-box styled, but each one for a specific feature (CRUD tests for example)

• Acceptance tests which should be completely black-box and are driven by functional requirements (as your users would phrase them). End-to-end as much as possible, and not knowing the internal of your chosen implementations. The textbook definition of black-box tests.

And remember code coverage is meaningless in most of the cases. You need a high lines coverage (or methods coverage, whatever your counting method is), but that's usually not sufficient. The concept you need to think about is functional coverage: making sure all your requirements and logical paths are covered.