Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Implementing `make check` or `make test`

How can I implement a simple regression test framework with Make? (I’m using GNU Make, if that matters.)

My current makefile looks something like this (edited for simplicity):

OBJS = jscheme.o utility.o model.o read.o eval.o print.o  %.o : %.c jscheme.h     gcc -c -o $@ $<  jscheme : $(OBJS)     gcc -o $@ $(OBJS)  .PHONY : clean  clean :     -rm -f jscheme $(OBJS) 

I’d like to have a set of regression tests, e.g., expr.in testing a “good” expression & unrecognized.in testing a “bad” one, with expr.cmp & unrecognized.cmp being the expected output for each. Manual testing would look like this:

$ jscheme < expr.in > expr.out 2>&1 $ jscheme < unrecognized.in > unrecognized.out 2>&1 $ diff -q expr.out expr.cmp # identical $ diff -q unrecognized.out unrecognized.cmp Files unrecognized.out and unrecognized.cmp differ 

I thought to add a set of rules to the makefile looking something like this:

TESTS = expr.test unrecognized.test  .PHONY test $(TESTS)  test : $(TESTS)  %.test : jscheme %.in %.cmp     jscheme < [something.in] > [something.out] 2>&1     diff -q [something.out] [something.cmp] 

My questions:
• What do I put in the [something] placeholders?
• Is there a way to replace the message from diff with a message saying, “Test expr failed”?

like image 903
J. C. Salomon Avatar asked Feb 07 '11 22:02

J. C. Salomon


People also ask

What is a make check?

make check is a command to a makefile. It does whatever the makefile defines it to do. It sounds like a little background on makefiles would be good. This is a tutorial that my school uses for a programming course.

What does make test do?

This command tests various parts of samba code base to ensure the correctness of the functionality. It uses the test framework built in samba code base to perform hundreds of tests. It does take a fair bit of time to execute all the tests.

How do I run a test in makefile?

...will run program.exe if you type "make testSomething" on your command line. And if you want to send the output from the program to yet another file... This way, if you type "make testSomething", your program will be executed, whenever it asks for input it will receive it from input.

What does $< mean in makefile?

The variable $@ represents the name of the target and $< represents the first prerequisite required to create the output file.


1 Answers

Your original approach, as stated in the question, is best. Each of your tests is in the form of a pair of expected inputs and outputs. Make is quite capable of iterating through these and running the tests; there is no need to use a shell for loop. In fact, by doing this you are losing the opportunity to run your tests in parallel, and are creating extra work for yourself in order to clean up temp files (which are not needed).

Here's a solution (using bc as an example):

SHELL := /bin/bash  all-tests := $(addsuffix .test, $(basename $(wildcard *.test-in)))  .PHONY : test all %.test  BC := /usr/bin/bc  test : $(all-tests)  %.test : %.test-in %.test-cmp $(BC)     @$(BC) <$< 2>&1 | diff -q $(word 2, $?) - >/dev/null || \     (echo "Test $@ failed" && exit 1)  all : test      @echo "Success, all tests passed." 

The solution directly addresses your original questions:

  • The placeholders you're looking for are $< and $(word 2, $?) corresponding to the prerequisites %.test-in and %.test-cmp respectively. Contrary to the @reinierpost comment temp files are not needed.
  • The diff message is hidden and replaced using echo.
  • The makefile should be invoked with make -k to run all the tests regardless of whether an individual test fails or succeeds.
  • make -k all will only run if all the tests succeed.

We avoid enumerating each test manually when defining the all-tests variable by leveraging the file naming convention (*.test-in) and the GNU make functions for file names. As a bonus this means the solution scales to tens of thousands of tests out of the box, as the length of variables is unlimited in GNU make. This is better than the shell based solution which will fall over once you hit the operating system command line limit.

like image 164
Richard Padley Avatar answered Oct 01 '22 07:10

Richard Padley