How can I capture testthat's error messages as part of the results in a dataframe?

Question

I'm using the testthat package to run tests, and I've been very happy with it so far. Currently, I'm using the test_file() function to run all tests and capture their results as a dataframe, and then do some additional analysis.

However, one thing that I would like to do is capture the error messages from failures in the dataframe itself. I really like how descriptive some of the messages are, and the extra information would be great to have downstream. Below is an example of what I'm thinking of. I would like to capture the "mean absolute difference: 1" message as a column in df.

> df <- test_file(f)
1

1. Failure(@testthat.R#4): insert me here --------------------------------------
1 not equal to expected
Mean absolute difference: 1

> df
        file context           test nb failed error  user system  real
1 testthat.R         insert me here  1      1 FALSE 0.004      0 0.004

The messages are getting generated, it doesn't seem like too much of a stretch to intercept those messages in one of the Reporter classes, but after poking around the docs and source, I don't think that functionality is built in.

So, is it possible to do this with the current version of testthat?

If not, what would need to happen to enable this feature? I am willing to contribute to the project, but I'm not sure where to start as I'm not very used to R's OOP.

Thanks for your input.

Answer 1

lentils, this is beyond my skill level too.

1. Have you tried posting this question to hadley on his github project testthat issues page. See Issue #215 he clearly wants to refactor the Reporter?

It may a Reporter they can add, or perhaps they can add it as a new field to the res dataframe in their test_results.R.

2. Have you tried the other reporters? I experimented with tap and it gives more compact and cleaner information to the display, which may help you more than the default.

Trying with my code (which is checking if code's name from database is correct for Apple share, and I deliberately added a misspelling):

> test_file("code/test-getdata-fun.R", reporter = "tap")
1..3
# Context Get Security Data Point 
ok 1 Inputs ok 
ok 2 Inputs ok 
not ok 3 Get Security Name OK 
  GetDataPoint(conn, test.apple.ticker, "NAME")$NAME not equal to "APPLE INCssss"
1 string mismatches:
x[1]: "APPLE INCssss"
y[1]: "APPLE INC"

3. Could you then capture this R display output to help you, i.e. using something like sink() to send output to a file and then search that looking for "not ok"s?

I realise this is not perfect, but it might be a workaround whilst the Reporter is being refactored.