How to perform regression tests in embedded systems

Question

What good practices and strategies are there for running regression tests in embedded environments or in other situations where the possibility to automate tests is very limited.

In my experience a lot of the testing has to be performed manually i.e. a tester needs to push a sequence of buttons and verify that the machine behaves correctly. As a developer it is really hard to assure yourself that your changes don't break something else.

Without proper regression tests the situation gets even worse during big refactorings and such.

Does anyone recognize the problem? Did you find a good solution or process to deal with this kind of problem?

Answer 1

For embedded testing, I would suggest that you design your way out of this very early in the development process. Sandboxing your embedded code to run on a PC platform helps a lot, and then do mocking afterwards :)

This will ensure integrety for the most of it, but you would still need to do system and acceptance testing manually later on.

Answer 2

Personally, I'm a big fan of having my embedded code compile on both the target hardware and my own computer. For example, when targeting an 8086, I included both an entry point that maps to reset on the 8086 hardware and a DOS entry point. The hardware was designed so all IO was memory mapped. I then conditionally compiled in a hardware simulator and conditionally changed the hardware memory locations to simulated hardware memory.

If I were to work on a non-x86 platform, I'd probably write an emulator instead.

Another approach is to create a test rig where all the inputs and outputs for the hardware are controlled through software. We use this a lot in factory testing.

One time we built a simulator into the IO hardware. That way the rest of the system could be tested by sending a few commands over CAN to put the hardware into simulated mode. Similarly, well-factored software could have a "simulated mode" where the IO is simulated in response to software commands.