testing command line utilities

Question

I'm looking for a way to run tests on command-line utilities written in bash, or any other language.

I'd like to find a testing framework that would have statements like

setup:
    command = 'do_awesome_thing'
    filename = 'testfile'
    args = ['--with', 'extra_win', '--file', filename]
    run_command command args

test_output_was_correct
    assert_output_was 'Creating awesome file "' + filename + '" with extra win.'

test_file_contains_extra_win
    assert_file_contains filename 'extra win'

Presumably the base test case would set up a temp directory in which to run these commands, and remove it at teardown.

I would prefer to use something in Python, since I'm much more familiar with it than with other plausible candidate languages.

I imagine that there could be something using a DSL that would make it effectively language-agnostic (or its own language, depending on how you look at it); however this might be less than ideal, since my testing techniques usually involve writing code that generates tests.

It's a bit difficult to google for this, as there is a lot of information on utilities which run tests, which is sort of the converse of what I'm looking for.

Support for doctests embedded in the output of command --help would be an extra bonus :)

Answer 1

Check out ScriptTest :

from scripttest import TestFileEnvironment

env = TestFileEnvironment('./scratch')

def test_script():
    env.reset()
    result = env.run('do_awesome_thing testfile --with extra_win --file %s' % filename)
    # or use a list like ['do_awesome_thing', 'testfile', ...]
    assert result.stdout.startswith('Creating awesome file')
    assert filename in result.files_created

It's reasonably doctest-usable as well.

Answer 2

Well... What we usually do (and one of the wonders of O.O. languages) is to write all the components of an application before actually make the application. Every component might have an standalone way to be executed, for testing purpose (command line, usually), that also allows you to think in them as complete programs each by each, and use them in future projects. If what you want is to test the integrity of an existing program... well, I think the best way is to learn in deep how it work, or even deeper: read the source. Or even deeper: develop a bot to force-test it :3

Sorry that's what I have .-.