How to combine unittest results using a Makefile?

Question

I want to use a Makefile to run individual test files or a combined version of all tests or a coverage report.

I'm pretty new to Makefiles so I borrowed one and adapted it. The result is here.

The problem is that make test will run each test in sequence and it is hard to see which ones failed when you have a bunch and the screen scrolls a lot. I like that each one uses a separate process so they don't interfere with each other, though.

Question is: can I combine the results more nicely using only the Makefile or I need a separate script? Do you know some good examples of Makefiles to run tests?

(I want to use Makefile + unittest + coverage only, and no other dependencies)

Answer 1

An alternative approach is to use unittest discovery, which will aggregate all your separate test files into a single run, e.g. in the Makefile

 test:
    python -m unittest discover -p '*tests.py' -v

If running the tests in parallel processes is important to you, then instead of using unittest to run the tests, use either nose or pytest. They each have options to run tests in parallel. You should be able to do this without any modifications to your test code.

Answer 2

Here is a quick hack you can insert into your Makefile with no changes to your infrastructure.

The special variable $? contains the exit status of the last command. Using it you can examine the return value of each test. In the script below I counted the number of tests that fail, and output that at the end of the run. You could also just exit immediately if one test fails so you wouldn't have to scroll up to see the output.

failed=0 \
for i in $(TESTS); \
do \
  echo $$i; \
  PYTHONPATH=$(GAEPATH):. $(PYTHON) -m tests.`basename $$i .py` $(FLAGS); \
  if [ $? -ne 0 ] \
  then \
   $failed=$(($failed+1)) \
  fi \
done \
if [$failed -ne 0] \
then \
  echo $failed Tests Failed \
  exit $failed \
fi \

There are definitely better and more robust ways of testing, but this hack should work if you so desire. I would suggest eventually moving the bash script below into python, and then all you would have to do is call ./run_tests.py to run all your unit tests. Since python is infinitely more expressive than bash, you have a lot more freedom to interpret and display the results. Depending on your needs, using a unit testing framework like unittest might be desirable to rolling your own code.