testing a program in bash

Question

I wrote a program in c++ and now I have a binary. I have also generated a bunch of tests for testing. Now I want to automate the process of testing with bash. I want to save three things in one execution of my binary:

1. execution time
2. exit code
3. output of the program

Right now I am stack up with a script that only tests that binary does its job and returns 0 and doesn't save any information that I mentioned above. My script looks like this

#!/bin/bash

if [ "$#" -ne 2 ]; then
    echo "Usage: testScript <binary> <dir_with_tests>"
    exit 1
fi

binary="$1"
testsDir="$2"

for test in $(find $testsDir -name '*.txt'); do
    testname=$(basename $test)

    encodedTmp=$(mktemp /tmp/encoded_$testname)
    decodedTmp=$(mktemp /tmp/decoded_$testname)

    printf 'testing on %s...\n' "$testname"

    if ! "$binary" -c -f $test -o $encodedTmp > /dev/null; then
        echo 'encoder failed'
        rm "$encodedTmp"
        rm "$decodedTmp"
        continue
    fi

    if ! "$binary" -u -f $encodedTmp -o $decodedTmp > /dev/null; then
        echo 'decoder failed'
        rm "$encodedTmp"
        rm "$decodedTmp"
        continue
    fi

    if ! diff "$test" "$decodedTmp" > /dev/null ; then
        echo "result differs with input"
    else
        echo "$testname passed"
    fi

    rm "$encodedTmp"
    rm "$decodedTmp"
done

I want save output of $binary in a variable and not send it into /dev/null. I also want to save time using time bash function

Answer 1

As you asked for the output to be saved in a shell variable, I tried answering this without using output redirection – which saves output in (temporary) text files (which then have to be cleaned).

Saving the command output

You can replace this line

if ! "$binary" -c -f $test -o $encodedTmp > /dev/null; then

with

if ! output=$("$binary" -c -f $test -o $encodedTmp); then

Using command substitution saves the program output of $binary in the shell variable. Command substitution (combined with shell variable assignment) also allows exit codes of programs to be passed up to the calling shell so the conditional if statement will continue to check if $binary executed without error.

You can view the program output by running echo "$output".

Saving the time

Without a more sophisticated form of Inter-Process Communication, there’s no way for a shell that’s a sub-process of another shell to change the variables or the environment of its parent process so the only way that I could save both the time and the program output was to combine them in the one variable:

if ! time-output=$(time "$binary" -c -f $test -o $encodedTmp) 2>&1); then

Since time prints its profiling information to stderr, I use the parentheses operator to run the command in subshell whose stderr can be redirected to stdout. The programming output and the output of time can be viewed by running echo "$time-output" which should return something similar to:

<program output>
<blank line>
real    0m0.041s
user    0m0.000s
sys     0m0.046s

Answer 2

You can get the process status in bash by using $? and print it out by echo $?. And to catch the output of time, you could use sth like that

{ time sleep 1 ; } 2> time.txt

Or you can save the output of the program and execution time at once

(time ls) > out.file 2>&1

Answer 3

You can save output to a file using output redirection. Just change first /dev/null line:

if ! "$binary" -c -f $test -o $encodedTmp > /dev/null; then

to

if ! "$binary" -c -f $test -o $encodedTmp > prog_output; then

then change second and third /dev/null lines respectively:

if ! "$binary" -u -f $encodedTmp -o $decodedTmp >> prog_output; then

if ! diff "$test" "$decodedTmp" >> prog_output; then

To measure program execution put

start=$(date +%s)

on the first line

then

end=$(date +%s)
echo "Execution time in seconds: " $((end-start)) >> prog_output

on the end.