Compiler Integration Tests in Haskell

Question

I am planning to write a little toy compiler in Haskell for a very simple language in order to reinforce my Haskell skills and have fun designing a new language. I am still thinking about some general decisions and one of the biggest open points is how I would do integration tests. I have the following requirements:

• It should be possible to create a list of triples (input, program, output), such that my integration test runs my compiler to compile the program, runs the compiled binary, passes the input to it and verifies that the output of the run is equal to the given output.
  • It should be possible to extend the integratiom test at a later point to test more complex interactions with the program, e.g. that it changes a file or sleeps for at least 20 seconds or something like that.

I also have the following optional requirements:

• It should be as much an "end to end" as possible, i.e. it should treat the wholencompiler as a blackbox as much as possible and it shouldn't have to access the internals of the compiler or anything like that.
• All code should be written in Haskell.
• It would be nice if I would get typical testing framework functionality for free, i.e. without implementing it myself. E.g. a green "SUCCESS" message or a collection of error messages describing failures.

I have tried to find something that fulfills my needs, but I wasn't successful so far. The alternatives I considered are the following:

• shunit would satisfy everything except the condition that I would like to write the code in Haskell.
• QuickCheck would allow me to write everything in Haskell, but as I understand it, it seems to be mostly suited for tests that involve just a Haskell function and its result. So I would need to test functions in the compiler and relax my "end to end" requirement.
• I could just write a Haskell program that starts the compiler in another process, passes it the input program and then starts the compiled code in another process, passes it the inpit and checks the output. This would however involve a lot of coding on my side in ordee to implement all the features that one gets for free when using a testing framework.

I am not sure yet which option I should choose and I still hope that I am missing a good solution. Do you have any idea on how I could create an integration test that fulfills all my requirements?

Answer 1

I'm thinking unsafePerformIO should be quite safe here considering the programs should never interact with anything the testing environment/logic depends on, or, in other words, the compilation and execution should be viewable as a pure function in the context of testing in a controlled, isolated environment. And I reckon it's indeed useful to test your compiler in such conditions. And QuickCheck should then become an option even for blackbox testing. But some clearer minds could prove me mistaken.

So assuming your compiler is smth like:

type Source = String

compile :: Source -> Program

and your program execution is

data Report = Report Output TimeTaken OtherStats ...

execute :: Program -> IO Report

you could quite safely use unsafePerformIO to convert that into

execute' :: Program -> Report -- or perhaps 'executeUnsafe'
execute' = unsafePerformIO . execute

and then

compileAndExec :: Source -> Report
compileAndExec = compile . execute'

and use that with QuickCheck.

---

Whether execute invokes a subprocess, obtains an actual binary, executes that, etc, — or interprets the binary (or bytecode) in-memory, is up to you.

But I'd recommend separating byte code and binary generation: this way you can test the compiler separately from the linker/whatnot, which is easier, and also get an interpreter in the process.