Test an emulator

Question

How should we organize unit tests and system tests for a program that emulates a hardware (that is like vmWare)?

Background:

We have for many years managed a computer from the 1980s and associated peripheral equipments and software. The system is critical to our customers and they do not want to replace it. We have therefore chosen to develop emulators for some of the hardware. The problem is that it is poorly documented in many thousands of pages of typewriter written text. It is therefore try and error-development.

The problem:

We currently have no unit tests of the emulator and the system tests are very add-hock. It is difficult to test whether a complex OS works in all aspects by typing in the text terminal and simulate data input from external systems. The only way we're testing right now is to add a large input pressure from external systems (via X.25) and automate some heavy operations regularly. But you miss so much then.

Answer 1

I used to work for a company doing something comparable. To test our system (which was actually a dynamic binary translator, not an emulator) we wrote a testing framework which would run the same commands natively and translated, then compare the results. We started doing this with userspace programs, but as we developed more sophisticated products we used the same techniques to automate testing of emulated hardware. Roughly speaking, you want to write some programs which access this hardware and do stuff to it, dumping all the output to the terminal or a log somewhere. Then run these programs on both sides (real and emulated), and compare the output. Depending on how exact your emulation is intended to be, you might need to some scripting to ignore certain parts of the output when diffing - addresses, hostnames, that sort of thing.

The other thing to watch out for when emulating hardware is state changes: a particular command might give the same output on both sides, but might change internal state in different ways. This can be tough to anticipate, but in general you need to identify internal state likely to be affected and dump it out along with output for each command.

Before we got bought we started working on something even cleverer, using kernel tracing tools to monitor OS/hardware state step-by-step as we ran our tests, then comparing the series of steps between native and translated runs. This was never fully developed, but looked very promising.

Sadly all this stuff was internal and closed-source, so I can't point you to anything you can run and play with, but the idea is very sound - we had thousands of automated tests like this running on every build of our translators with very satisfactory results.

Edit: The more I think about this problem the more keen I am to have a whack at it. I don't suppose your project is open-source, but if it is I'd love to participate. Feel free to contact me if that's a possibility.