Bash script to automatically test program output - C

Question

I am very new to writing scripts and I am having trouble figuring out how to get started on a bash script that will automatically test the output of a program against expected output.

I want to write a bash script that will run a specified executable on a set of test inputs, say in1 in2 etc., against corresponding expected outputs, out1, out2, etc., and check that they match. The file to be tested reads its input from stdin and writes its output to stdout. So executing the test program on an input file will involve I/O redirection.

The script will be invoked with a single argument, which will be the name of the executable file to be tested.

I'm having trouble just getting going on this, so any help at all (links to any resources that further explain how I could do this) would be greatly appreciated. I've obviously tried searching myself but haven't been very successful in that.

Thanks!

Answer 1

If I get what you want; this might get you started:

A mix of bash + external tools like diff.

#!/bin/bash

# If number of arguments less then 1; print usage and exit
if [ $# -lt 1 ]; then
    printf "Usage: %s <application>\n" "$0" >&2
    exit 1
fi

bin="$1"           # The application (from command arg)
diff="diff -iad"   # Diff command, or what ever

# An array, do not have to declare it, but is supposedly faster
declare -a file_base=("file1" "file2" "file3")

# Loop the array
for file in "${file_base[@]}"; do
    # Padd file_base with suffixes
    file_in="$file.in"             # The in file
    file_out_val="$file.out"       # The out file to check against
    file_out_tst="$file.out.tst"   # The outfile from test application

    # Validate infile exists (do the same for out validate file)
    if [ ! -f "$file_in" ]; then
        printf "In file %s is missing\n" "$file_in"
        continue;
    fi
    if [ ! -f "$file_out_val" ]; then
        printf "Validation file %s is missing\n" "$file_out_val"
        continue;
    fi

    printf "Testing against %s\n" "$file_in"

    # Run application, redirect in file to app, and output to out file
    "./$bin" < "$file_in" > "$file_out_tst"

    # Execute diff
    $diff "$file_out_tst" "$file_out_val"


    # Check exit code from previous command (ie diff)
    # We need to add this to a variable else we can't print it
    # as it will be changed by the if [
    # Iff not 0 then the files differ (at least with diff)
    e_code=$?
    if [ $e_code != 0 ]; then
            printf "TEST FAIL : %d\n" "$e_code"
    else
            printf "TEST OK!\n"
    fi

    # Pause by prompt
    read -p "Enter a to abort, anything else to continue: " input_data
    # Iff input is "a" then abort
    [ "$input_data" == "a" ] && break

done

# Clean exit with status 0
exit 0

---

Edit.

Added exit code check; And a short walk trough:

This will in short do:

1. Check if argument is given (bin/application)
2. Use an array of "base names", loop this and generate real filenames.
   • I.e.: Having array ("file1" "file2") you get
     
     • In file: file1.in
     • Out file to validate against: file1.out
     • Out file: file1.out.tst
     • In file: file2.in
     • ...
3. Execute application and redirect in file to stdin for application by <, and redirect stdout from application to out file test by >.
4. Use a tool like i.e. diff to test if they are the same.
5. Check exit / return code from tool and print message (FAIL/OK)
6. Prompt for continuance.

Any and all of which off course can be modified, removed etc.

---

Some links:

• TLDP; Advanced Bash-Scripting Guide (can be a bit more readable with this)
  • Arrays
  • File test operators
  • Loops and branches
  • Exit-status
  • ...
• bash-array-tutorial
• TLDP; Bash-Beginners-Guide

Answer 2

Expect could be a perfect fit for this kind of problem:

Expect is a tool primarily for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc. Expect really makes this stuff trivial. Expect is also useful for testing these same applications.