How to test a Command Line Interface (CLI)?

Question

My Java application consists of two parts:

1. core libraries (classes, interfaces, etc)
2. command line interface (CLI), which uses the core libraries

For 1. I use JUnit for unit testing, but what would you do for 2.?

How can I create automated tests for a command line interface?

Answer 1

For the CLI, try BATS (Bash Automated Testing System): https://github.com/bats-core/bats-core

From the docs:

example.bats contains
#!/usr/bin/env bats

@test "addition using bc" {
  result="$(echo 2+2 | bc)"
  [ "$result" -eq 4 ]
}  

@test "addition using dc" {
  result="$(echo 2 2+p | dc)"
  [ "$result" -eq 4 ]
}


$ bats example.bats

 ✓ addition using bc
 ✓ addition using dc

2 tests, 0 failures

bats-core

Answer 2

I had the exact same problem, landed here and didn't find a good answer, so I thought I would post the solution I eventually came to as a starting point for anyone who lands here in the future.

I wrote my tests after the CLI (shame on me, I know), so first I made sure the CLI was written in a testable way. It looks something like this (I've omitted the exception handling and simplified a lot to make it more readable):

public class CLI {

    public static void main(String... args) {
        new CLI(args).startInterface();
    }

    CLI(String... args) {
        System.out.println("Welcome to the CLI!");
        // parse args, load resources, etc
    }

    void startInterface() {
        BufferedReader consoleReader = new BufferedReader(new InputStreamReader(System.in));
        while (true) {
            String[] input = sanitiseInput(consoleReader.readLine());

            if (input[0].equalsIgnoreCase("help") {
                help();
            } else if (input[0].equalsIgnoreCase("exit") {
                break;
            } else if (input[0].equalsIgnoreCase("save") {
                save(input);
            } else {
                System.out.println("Unkown command.");
            }
        }
    }

    String[] sanitiseInput(String rawInput) {
        // process the input and return each part of it in order in an array, something like:
        return rawInput.trim().split("[ \t]+");
    }

    void help() {
        // print help information
        System.out.println("Helpful help.");
    }

    void save(String[] args) {
        // save something based on the argument(s)
    }
}

On to testing. CLI is not a part of the public libraries, so it should be protected from library users. As is mentioned here, you can use the default access modifier to make it package private. This gives your tests full access to the class (as long as they are in the same package) while still protecting it, so that's that taken care of.

Writing a method for each command accepted by the CLI allows JUnit tests to almost perfectly simulate user input. Since the object won't read from stdin until you call startInterface(), you can simply instantiate it and test the individual methods.

First, it's good to test that the raw input is being correctly sanitised, which you can do trivially by writing JUnit tests for sanitiseInput(). I wrote tests like this:

@Test
public void commandAndArgumentsSeparatedBySpaces() throws Exception {
    String[] processedInput = uut.sanitiseInput("command argument1 argument2");

    assertEquals("Wrong array length.", 3, processedInput.length);
    assertEquals("command", processedInput[0]);
    assertEquals("argument1", processedInput[1]);
    assertEquals("argument2", processedInput[2]);
}

It's easy to cover some edge cases too:

@Test
public void leadingTrailingAndIntermediaryWhiteSpace() throws Exception {
    String[] processedInput = uut.sanitiseInput("  \t  this   \twas \t  \t  a triumph  \t\t    ");

    assertEquals("Wrong array length.", 4, processedInput.length);
    assertEquals("this", processedInput[0]);
    assertEquals("was", processedInput[1]);
    assertEquals("a", processedInput[2]);
    assertEquals("triumph", processedInput[3]);
}

Next we can test the invididual command methods by monitoring stdout. I did this (which I found here):

private CLI uut;
private ByteArrayOutputStream testOutput;
private PrintStream console = System.out;
private static final String EOL = System.getProperty("line.separator");

@Before
public void setUp() throws Exception {
    uut = new CLI();
    testOutput = new ByteArrayOutputStream();
}

@Test
public void helpIsPrintedToStdout() throws Exception {
    try {
        System.setOut(new PrintStream(testOutput));
        uut.help();
    } finally {
        System.setOut(console);
    }
    assertEquals("Helpful help." + EOL, testOutput.toString());
}

In other words, substitute the JVM's out with something you can query just before the exercise, and then set the old console back in the test's teardown.

Of course, CLI applications often do more than just print to the console. Supposing your program saves information to a file, you could test it as such (as of JUnit 4.7):

@Rule
public TemporaryFolder tempFolder = new TemporaryFolder();

@Test
public void informationIsSavedToFile() throws Exception {
    File testFile = tempFolder.newFile();
    String expectedContent = "This should be written to the file.";

    uut.save(testFile.getAbsolutePath(), expectedContent);

    try (Scanner scanner = new Scanner(testFile)) { 
        String actualContent = scanner.useDelimiter("\\Z").next();
        assertEquals(actualContent, expectedContent);
    }
}

JUnit will take care of creating a valid file and removing it at the end of the test run, leaving you free to test that it is properly treated by the CLI methods.