Component vs Integration vs Functional tests

Question

Recently I figured out that my understanding of different types of tests maybe is not completely right.

E.g. Unit test is testing of one unit where interaction with other units is based on mocks (fakes, stubs). So, no interaction with file system, threads, time...

Component tests, for me, were tests around one component (more units) where I used both, mocks and "real" resources. Both of them I used for input simulation and output tests. Whatever seemed more appropriate. E.g. I am mocking change of current arbitration state, but I am asserting that events are stored into RTDB.

These components, for me, were usually slices of one application.

Functional tests I considered as (black box) tests around my application (exe) that runs in production.

Well, is this really true or not? Are component tests only based on mocks? If yes, why? How can I be sure that mocks are good enough? Should we run application from functional tests or not? Why is it different than bootstrap of app main routine in a thread? What are integration tests?

I would like to hear other opinions and how do you do that. What tests do you have, how do you maintain them and who is responsible for them in your team?

Cheers!

Answer 1

Your question is somewhat unfocused, but I will nevertheless try to give an answer.

Various kinds of tests have already been discussed here: What is Unit test, Integration Test, Smoke test, Regression Test?

It is, however, not primarily the use of doubles (like mocks, stubs etc.) which distinguishes the different kinds of tests. It is the goal that you pursue with the different tests that distinguishes them.

In unit-testing, the goal is to test that the piece of code that you write behaves as you expect it to behave. That is, you are testing your code against your assumptions of how the code should behave. Doubling (mocking) is only a means to achieve this goal while at the same time ensuring a) testing of all scenarios (including error cases) b) deterministic test results (independent from timing / scheduling / environment conditions) c) fast test build and execution d) easy test setup e) independence from problems with libraries (incomplete, buggy, ...). You don't have to double a library if you can achieve all these criteria with the use of the real library. For example, in most cases you don't create doubles for container libraries (lists, sets, maps, ...) that are part of the standard library of your programming language - you will typically reach all your unit-test goals by just using the library.

It is not a goal of unit-testing to identify misconceptions about how to interact with other software parts. And, since it is not mandatory but perfectly fine to double software parts your unit interacts with, you will in this case not even be able to identify such misconceptions: Whenever you double one of your dependencies, you are implementing the double according to your understanding (and, possibly, your misconceptions) of the other component. Thus, the identification of misconceptions about how to interact with other components is the goal of integration testing. In integration testing you bring together those components, the interactions between which you want to test. Additional components, again, can be linked in or doubled, depending on similar criteria as with unit-testing.

Once you realize the different goals of unit-tests and integration-tests you will also realize that the different goals will lead to the construction of different sets of test cases. Therefore, a unit-test suite remains a unit-test suite, no matter if you can get along with linking the real libraries instead of using doubles.

Then, (sub-)system-testing brings in different goals again, for example to test that a set of components together actually implement a required feature of that (sub-)system. An example could be that components A and B operate on a list of structures, and that among other things, the final list shall be sorted. Now, the developers of A and B might assume that the other would be doing the sorting. And, if neither of them require the data to be sorted for their own code, a) both components will have unit-test suites that don't test for the output being sorted, and b) there will be integration tests for the interaction between A and B that don't necessarily test that the input from the other comes sorted. With (sub-)system testing, however, the existence of all required features will be tested and the missing sort will be detected. Again, additional components in the environment of the (sub-)system test can be linked in or doubled - both scenarios are possible.

You are, however, asking about component-testing, which will most likely either be a synonym for (sub-)system testing, or a term describing a combination of (sub-)system testing and the integration tests for the components that form this (sub-)system. And, the functional tests you mention are most likely (sub-)system tests on the level of your software system (therefore, not sub- but just system tests). And, finally answering the question as I understand it, it should have become clear now that the use of doubles is not a reliable criterion to classify the test suites, but the respective goal of the tests is.