How to Use `bats-mock` to Assert Against Calls to a Mocked Script in Bash Testing

Question

I'm trying to use bats to test some critical shell scripts in a project I'm working on. I'd like to be able to mock scripts out in order to assert that a script calls another script with the correct arguments in a given situation. The bats-mock library seems like it should do the trick, but it hasn't been documented at all.

I've tried looking at the bats-mock code and several test helper scripts other people have created (like this one), but unfortunately I'm not comfortable enough with bash to be able to deduce how to correctly use the bats-mock library.

How can I use the bats-mock library to mock out a script and assert against calls to the mock?

Answer 1

Brief suggestion:

There is a newer more actively developer bats-mock that uses a slightly different approach that could be worth exploring. https://github.com/grayhemp/bats-mock

I'll be back later with.....MORE.

---

Back with more:

The main difference between them is which 'test double' style they implement. Brief explanation of some styles per Martin Fowler quoting a book covering many testing strategies, in his article mocksArentStubs

Meszaros uses the term Test Double as the generic term for any kind of pretend object used in place of a real object for testing purposes. The name comes from the notion of a Stunt Double in movies. (One of his aims was to avoid using any name that was already widely used.) Meszaros then defined four particular kinds of double:

• Dummy objects are passed around but never actually used. Usually they are just used to fill parameter lists.
• Fake objects actually have working implementations, but usually take some shortcut which makes them not suitable for production (an in memory database is a good example).
• Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what's programmed in for the test.
• Spies are stubs that also record some information based on how they were called. One form of this might be an email service that records how many messages it was sent.
• Mocks are what we are talking about here: objects pre-programmed with expectations which form a specification of the calls they are expected to receive.

Of these kinds of doubles, only mocks insist upon behavior verification. The other doubles can, and usually do, use state verification. Mocks actually do behave like other doubles during the exercise phase, as they need to make the SUT believe it's talking with its real collaborators - but mocks differ in the setup and the verification phases.

JasonKarns' appears to be primarily designed around enabling stubs where it returns dummy data for N number of calls to a script or binary and keeps an internal count of the number of calls to the stub and returns an error code if the calls don't match the N lines of fake data.

Grayhemp's version allows you to create a spy object and the output it should produce as well as its return code and any "side effects" that should be triggered when the mock is run (like a PID in the example below). You can then run a script or command that calls the command that the mock is shadowing and see how many times it was called and what the return code of the script was as well as the environment present when the mock command was called. Overall it seems to make it easier to assert what a script/binary was called with and how many times it was called.

BTW, if you too wasted hours wondering what in the $)*#@$ would the code for get_timestamp look like in the jasonkarns example or what are the ${_DATE_ARGS}, here's my best guess based on the examples in this other answer for getting time since the epoch in milliseconds, https://serverfault.com/a/588705/266525:

You can copy and paste this into a Bash/POSIX shell to see that the first output matches what the first stub data line would be giving to get_timestamp and the second output matches the output that the first assert in the bats-mock example shows.

get_timestamp () {
  # This should really be named get timestamp in milliseconds

  # In truth it wouldn't accept input ie the ${1} below,
  # but it is easier to show and test how it works with a fixed date (which is why we want to stub!)
  GIVEN_DATE="$1"
  # Pass in a human readable date and get back the epoch `%s` (seconds since 1-1-1970) and %N nanoseconds
  # date +%s.%N -d'Mon Apr 18 03:19:58.184561556 CDT 2016'
  EPOCH_NANO=$(date +%s.%N -d"$GIVEN_DATE")
  echo "This reflects the data the date stub would return: $EPOCH_NANO"
  # Accepts input in seconds.nanoseconds ie %s.%N and
  # sets the output format to milliseconds,
  # by combining the epoch `%s` (seconds since 1-1-1970) and
  # first 3 digits of the nanoseconds with %3N
  _DATE_ARGS='+%s%3N -d'
  echo $(date ${_DATE_ARGS}"@${EPOCH_NANO}")
}
get_timestamp 'Mon Apr 18 03:19:58.184561556 CDT 2016' # The quotes make it a *single* argument $1 to the function

Example from jasonkarns/bats-mock docs, note to the left of the : are the incoming arguments required for the stub to match, if you called date with different arguments it may pass through and hit the real thing, but I haven't tested this since I already spent WAY too much time figuring out the original function in order to give a better comparison to the other bats-mock implementation.

# In bats you can declare globals outside your tests if you want them to apply
# to all tests in a file, or in a `fixture` or `vars` file and `load`or `source` it
declare -g _DATE_ARGS='+%s.%N -d'

# The interesting thing about the order of the mocked call returns is they are actually moving backwards in time,
# very interesting behavior and possibly needs another test that should throw a really big exception if this is encountered in the real world

# Original example below
@test "get_timestamp" {
  stub date \
      "${_DATE_ARGS} : echo 1460967598.184561556" \
      "${_DATE_ARGS} : echo 1460967598.084561556" \
      "${_DATE_ARGS} : echo 1460967598.004561556" \
      "${_DATE_ARGS} : echo 1460967598.000561556" \
      "${_DATE_ARGS} : echo 1460967598.000061556"

  run get_timestamp
  assert_success
  assert_output 1460967598184

  run get_timestamp
  assert_success
  assert_output 1460967598084

  run get_timestamp
  assert_success
  assert_output 1460967598004

  run get_timestamp
  assert_success
  assert_output 1460967598000

  run get_timestamp
  assert_success
  assert_output 1460967598000

  unstub date
}

Example from grayhemp/bats-mock's README, note the cool mock_set-* and mock_get_* options.

@test "postgres.sh starts Postgres" {
  mock="$(mock_create)"
  mock_set_side_effect "${mock}" "echo $$ > /tmp/postgres_started"

  # Assuming postgres.sh expects the `_POSTGRES` variable to define a
  # path to the `postgres` executable
  _POSTGRES="${mock}" run postgres.sh

  [[ "${status}" -eq 0 ]]
  [[ "$(mock_get_call_num ${mock})" -eq 1 ]]
  [[ "$(mock_get_call_user ${mock})" = 'postgres' ]]
  [[ "$(mock_get_call_args ${mock})" =~ -D\ /var/lib/postgresql ]]
  [[ "$(mock_get_call_env ${mock} PGPORT)" -eq 5432 ]]
  [[ "$(cat /tmp/postgres_started)" -eq "$$" ]]
}

To get very similar behavior to jasonkarns version you need to inject the stub (aka symlink to the ${mock}) into the PATH yourself before calling the function. If you do this in the setup() method it occurs for every test, which might not be what you want, and you'll also want to make sure you remove the symlink in the teardown(), otherwise you can do the stubbing within the test and cleanup right at the end of the test (similar to the stub/unstub of jasonkarns version), though if you do it often you'll want to make it a test helper (basically reimplementing jasonkarns/bats-mock's stub inside grayhemp/bats-mock) and keep the helper with your tests so you can load or source it and reuse the functions in many tests. Or you could submit a PR to grayhemp/bats-mock to include the stubbing functionality (the race for DigitalOcean Hacktoberfest fame and infamy is on, and don't forget there is swag involved too!).

@test "get_timestamp" {
  mocked_command="date"
  mock="$(mock_create)"
  mock_path="${mock%/*}" # Parameter expansion to get the folder portion of the temp mock's path
  mock_file="${mock##*/}" # Parameter expansion to get the filename portion of the temp mock's path
  ln -sf "${mock_path}/${mock_file}" "${mock_path}/${mocked_command}"
  PATH="${mock_path}:$PATH" # Putting the stub at the beginning of the PATH so it gets picked up first
  mock_set_output "${mock}" "1460967598.184561556" 1
  mock_set_output "${mock}" "1460967598.084561556" 2
  mock_set_output "${mock}" "1460967598.004561556" 3
  mock_set_output "${mock}" "1460967598.000561556" 4
  mock_set_output "${mock}" "1460967598.000061556" 5
  mock_set_status "${mock}" 1 6

  run get_timestamp
  [[ "${status}" -eq 0 ]]
  run get_timestamp
  run get_timestamp
  run get_timestamp
  run get_timestamp
  [[ "${status}" -eq 0 ]]
  # Status is just of the previous invocation of `run`, so you can test every time or just once
  # note that calling the mock more times than you set the output for does NOT change the exit status...
  # unless you override it with `mock_set_status "${mock}" 1 6` 
  # Last bits are the exit code/status and index of call to return the status for
  # This is a test to assert that mocked_command stub is in the path and points the right place
  [[ "$(readlink -e $(which date))" == "$(readlink -e ${mock})" ]]
  # This is a direct call to the stubbed command to show that it returns the `mock_set_status` defined code and shows up in the call_num 
  run ${mocked_command}
  [[ "$status" -eq 1 ]]
  [[ "$(mock_get_call_num ${mock})" -eq 6 ]]
  # Check if your function exported something to the environment, the example get_timestamp function above does NOT
  # [[ "$(mock_get_call_env ${mock} _DATE_ARGS 1)" -eq '~%s%3N' ]]

  # Use the below line if you actually want to see all the arguments the function used to call the `date` 'stub'
  # echo "# call_args: " $(mock_get_call_args ${mock} 1) >&3

  # The actual args don't have the \ but the regex operator =~ treats + specially if it isn't escaped
  date_args="\+%s%3N"
  [[ "$(mock_get_call_args ${mock} 1)" =~ $date_args ]]

  # Cleanup our stub and fixup the PATH
  unlink "${mock_path}/${mocked_command}"
  PATH="${PATH/${mock_path}:/}"
}

If anybody needs more clarification or wants to have a working repository let me know and I can push my code up.