regression testing the entire app in Python

Question

I have a small command-line application (about 6k lines). It has no unit tests because I didn't know how to write them; but I'm retroactively adding some now. I read this tutorial but I'm left puzzled about how to test the whole application using this module; in fact, I'm not even sure if what I want to do is called a "unit test".

Specifically, if I run my application with certain parameters, it is supposed to generate certain output files. I want to make sure those output files are unchanged.

Namely, the following command-line invocations of my application:

main.py config1.txt 100 15
main.py config2.txt def 10 qa
etc.....

create a few small output text files (< 10 MB each) and place them into individual folders (one each per invocation), named such:

output/config1.100.15.201202011733/
output/config2.def.10.qa.201202011733/
etc...

Each folders contains a few small text files (<10MB each). After each iteration of code changes, I'd like to run my application with a few dozen command line parameters, and note any cases where the output files differ. (Ideally, I'd like to do more than that; e.g., for some output files, compare them as tab-separated tables with a certain primary key, so that if row order changed, they will still evaluate as equal; but that's not critical).

What's a good way to set this up?

Answer 1

Step 1. Break your app into two pieces.

1. The piece that uses optparse (or argparse) to parse the command-line options.

2. The piece that does the real work.

Your "main" script then does part 1 to get all the options and invokes part 2 to do the real work.

This is called "design for testability" and is the more important part of unit testing.

Step 2. Once you have two pieces, test the part that does the real work.

Write unit test scripts that from the_app import the_function_or_class_that_does_real_work

Test that function or class or whatever that does the real work.

class Test_With_File( TestCase ):
    def assertFileMatches( self, expected, actual, error=None ):
        # use difflib or whatever to compare the two files.

class TestMain_Sample_File1( Test_With_File ):
    def test_should_produce_known_output( self ):
        with open("temp","w") as target:
            the_function_that_does_real_work( 'config1.txt', arg2=100, arg3=15, out=target )
        self.assertFileMatches( "output/config1.100.15.201202011733", "temp" )

You can write variations on the "TestMain_Sample_File1" class to cover as many test cases as you find interesting.