Does there exist an established standard for testing command line arguments?

Question

I am developing a command line utility that has a LOT of flags. A typical command looks like this:

mycommand --foo=A --bar=B --jar=C --gnar=D --binks=E

In most cases, a 'success' message is printed but I still want to verify against other sources like an external database to ensure actual success.

I'm starting to create integration tests and I am unsure of the best way to do this. My main concerns are:

1. There are many many flag combinations, how do I know which combinations to test? If you do the math for the 10+ flags that can be used together...
2. Is it necessary to test permutations of flags?
3. How to build a framework capable of automating the tests and then verifying results.
4. How to keep track of a large number of flags and providing an order so it is easy to tell what combinations have been implemented and what has not.

The thought of manually writing out individual cases and verifying results in a unit-test like format is daunting.

Does anyone know of a pattern that can be used to automate this type of test? Perhaps even software that attempts to solve this problem? How did people working on GNU commandline tools test their software?

Answer 1

I think this is very specific to your application.

First, how do you determine the success of the execution of you application? Is it a result code? Is it something printed to the console?

For question 2, it depends how you parse those flags in your application. Most of the time, order of flags isn't important, but there are cases where it is. I hope you don't need to test for permutations of flags, because it would add a lot of cases to test.

In a general case, you should analyse what is the impact of each flag. It is possible that a flag doesn't interfere with the others, and then it just need to be tested once. This is also the case for flags that are meant to be used alone (--help or --version, for example). You also need to analyse what values you should test for each flag. Usually, you want to try each kind of possible valid value, and each kind of possible invalid values.

I think a simple bash script could be written to perform the tests, or any scripting language, like Python. Using nested loops, you could try, for each flag, possibles values, including tests for invalid values and the case where the flag isn't set. I will produce a multidimensional matrix of results, that should be analysed to see if results are conform to what expected.

Answer 2

When I write apps (in scripting languages), I have a function that parses a command line string. I source the file that I'm developing and unit test that function directly rather than involving the shell.