How to test for multiple warnings in an unknown order using testthat?

Question

I want to test that a function generates multiple warnings (4 or more), when the order of the warnings can vary. My best attempt to do this is based on look-ahead RegExp matching. Simplifying to just 2 warnings, I know my RegExp work on a single string output, because both of the following are true:

grepl("(?s)(?=.*2)(?=.*1)", "* warn 1.\n* warn 2.", perl=TRUE)
grepl("(?s)(?=.*2)(?=.*1)", "* warn 2.\n* warn 1.", perl=TRUE)

However, this does not work when testing multiple warnings with testhat::expect_warning

# The function generating warnings:
foo <- function() { warning("warn 1."); warning("warn 2.") }
foo()
Warning messages:
1: In foo() : warn 1.
2: In foo() : warn 2.

# Testing it
expect_warning( foo(), "(?s)(?=.*1)(?=.*2)", perl=TRUE)

Error: foo() does not match '(?s)(?=.*1)(?=.*2)'. Actual values:
* warn 1.
* warn 2.

I suspect this is because the internals of expect_warning are doing something like testing the given RegExp against each warning separately--why an expect_warning( ... all=TRUE ) parameter might be meaningful.

Unfortunately I can't use this with a RegExp like "1 | 2"; that succeeds if only one warning is given.

I also want to avoid running the function multiple times and testing a different warning each time. There is a large amount of setup and teardown code needed to test the real function. It interacts heavily with the file system, and since it is the file-system warnings I am testing for, I can't mock that. Moreover, I want to test for the warnings in multiple situations, each of which requires different setup and teardown code, so this quickly bloats my tests.

Any suggestion on how to test for multiple warnings simply and all at once?

Answer 1

To capture the warnings and analyze them "by hand", you can also use testthat::capture_warnings:

# The function generating warnings:
foo <- function() { warning("warn 1."); warning("warn 2.") }

w <- capture_warnings(foo())
expect_match(w, ".*1", all = FALSE)
expect_match(w, ".*2", all = FALSE)
expect_match(w, ".*3", all = FALSE)

(The last line raises an error.)

Answer 2

So I have cobbled together a solution that involves using my own warning catch loop and checking the warnings as strings. Not an "all at once" solution, but it is at least compact given a complex function call. My code as it would apply to the example function foo() is the following:

gotWarnings= character(0)
withCallingHandlers({
   got <- foo()
   }, warning= function(e) {
      # Push warning onto vector in parent frame.
      gotWarnings <<- c(gotWarnings, conditionMessage(e))
      invokeRestart("muffleWarning")
})

# Ensure no unexpected warnings, 
expect_equal(length(gotWarnings), 2)

# Test that each warning I want is there
expect_true( any( grepl( "warn 1\\.", gotWarnings )))
expect_true( any( grepl( "warn 2\\.", gotWarnings )))

Figuring out how to continue processing after catching a warning was the hard part; tryCatch exits after catching the first warning. Here and here helped me work this out. Perhaps this could be converted into an expect_all_warnings test that takes a vector of RegEx to match, maybe with all= TRUE meaning no warning can go unmatched. I'll hold off accepting this as an answer in case somebody posts a better one, or at least one that packages a solution like this nicely.