Why is the GHC test suite written in Python, not Haskell?

Question

I noticed that GHC (a widely-used Haskell compiler) has a test suite written in Python, not in Haskell (as I would naively expect). What is the history of this? Are there particular advantages to writing the test suite in a different language?

edit: Per a suggestion in the comments, I asked this in /r/haskell. It has now generated three answers, which I've quoted below:

tathougies said:

The test suite driver seems to be written in Python. Python is a good high-level scripting language.

It's like asking 'why does GHC use Make instead of haskell'? Probably because make is better at running shell programs with external dependency resolution built-in.

The tests themselves seem to be written in Haskell, verifying certain properties of the compiler and catching regressions. If they fail, it looks like the python driver is informed, and then would report the error to the user.

phadej added:

FWIW GHC's built system is being rewritten to use shake: the Haskell library.

eacameron said:

I don't know. But GHC doesn't have the luxury of using Haskell the same way you and I do. It has to bootstrap using a previous version of itself and it wants to avoid dependencies. Python is a pretty light-weight requirement since most systems (except Windows) come with it built in.

Answer 1

The commit message introducing Python explains a lot of it:

Revamp the testsuite framework. The previous framework was an experiment that got a little out of control - a whole new language with an interpreter written in Haskell was rather heavyweight and left us with a maintenance problem.

So the new test driver is written in Python. The downside is that you need Python to run the testsuite, but we don't think that's too big a problem since it only affects developers and Python installs pretty easily onto everything these days.

Highlights:

• 790 lines of Python, vs. 5300 lines of Haskell + 720 lines of <strange made-up language>.

• the framework supports running tests in various "ways", which should catch more bugs. By default, each test is run in three ways: normal, -O, and -O -fasm. Additionally, if profiling libraries have been built, another way (-O -prof -auto-all) is added. I plan to also add a 'GHCi' way.

  Running tests multiple ways has already shown up some new bugs!

• documentation is in the README file and is somewhat improved.

• the framework is rather less GHC-specific, and could without much difficulty be coaxed into using other compilers. Most of the GHC-specificness is in a separate configuration file (config/ghc).

Things may need a while to settle down. Expect some unexpected failures.