How do I test a Ruby command-line program with Cucumber that has a massive amount of output?

Question

I'm building a Ruby command-line program, and using Cucumber and Aruba to test it. Aruba includes some really handy matchers, so I can test output with a few lines in the .feature file:

When I run `myprogram`
Then it should pass with:
  """
  my program output
  """

The problem is that my program may contain dozens or even hundreds of lines of output; putting all that in the .feature file will make it harder to read and navigate (and is kind of obnoxious). What's the recommended way to test the output in such a case?

Answer 1

The short answer is: you should not do that.

Cucumber tests are supposed to be user-facing and readable. They describe features. A user isn't going to care if error output matches some known value byte for byte.

You need to ask yourself: What am I testing? Is the answer the error message? Probably not. You're testing some functionality in your application. If all you really want to ensure it fails, then what you want in your Cucumber scenario is the following line:

Then the exit status should not be 0

This assumes the script follows the standard convention that a non-zero exit status signals an error.

If your scenario requires that there be a certain message in the output, you can add it:

Then it should fail with
"""
Some error message
"""

but this need not be the entire output, only a partial match. (Note that there's a "it should fail with exactly:" defined in aruba, but I don't recommend using it.)

Edit: You've changed your example to be testing for a pass rather than a fail, but my basic advice is the same:

1. Separate the specifics of the output from the logic of the scenario. Using the example in the comments, if you have a test that confirms you can output a single user-generated comment, and another test that confirms you have output the correct 100 comments, then it is sufficient, you don't need to have 100 comments' worth of output in the Cucumber scenario.
2. Keep the Cucumber scenarios written from the user's perspective. Each scenario should test something that will be important to the user. Try to keep them minimal by removing anything that leaks in from the implementation when the user wouldn't care.
3. Use the built-in Aruba constructs that test for partial matches to achieve this. Look for key words or phrases in the output. Not only will the Cucumber tests be easier to read, they'll be more robust and immune to unrelated output changes.

Answer 2

To test your program you have 2 options:

1. Check only relevant part of the message (what you actually want to check with this test). Aruba has built-in stepdef for it so it's very easy

2. Check full message. If message is short, you can use Aruba's built-in stepdef. However, if message is long, you may like to put it in a separate file. As Aruba doesn't contain such method, you should write this stepdef yourself.

It may look like:

# Require aruba/api before that
Then /^it should (pass|fail) with message from file "(.*)"$/ do |pass_fail, filename|
  exact_output = File.read(filename)
  Aruba::API::assert_exit_status_and_output(pass_fail == "pass", exact_output, true)
end

If you want to write a lot of tests and a lot of messages are similar, you may end up with a lot of WET messsages. Asserting full message will make supporting tests harder in case of changes.

So you may want to use some sort of template engine to assert those messages to make tests DRYer.

You could design your tests differently.
If you use only first method, then your tests may not find some regression bugs as they don't test everything. If you'll use second method, it may be much more difficult to support them in case if there are a lot of repetitions between messages.

So there is a trade-off between those two options. Usually you build some tests with method 1, other one with method 2. You should think about what will be better in your case. I don't know any golden rule.