testthat fails within devtools::check but works in devtools::test

Question

Is there any way to reproduce the environment which is used by devtools::check?

I have the problem that my tests work with devtools::test() but fail within devtools::check(). My problem is now, how to find the problem. The report of check just prints the last few lines of the error log and I can't find the complete report for the testing.

checking tests ... ERROR
Running the tests in ‘tests/testthat.R’ failed.
Last 13 lines of output:
...

I know that check uses a different environment compared to test but I don't know how I should debug these problems since they are not reproducible at all. Specially these test where running a few month ago, so not sure where to look for the problem.

EDIT

actually I tried to locate my problem and I found a solution. But to post my solution to it, I have to add more details.

So my test always failed since I was testing a markdown script if it is running without errors and afterwards I was checking if some of the environmental variables are set correctly. These where results which I calculate with the script as well as standard settings which I set. So I wanted to get a warning if I forgot to change some of my settings after developing...

Anyway, since it is a markdown script, I had to extract the code and I was using comments from this post knitr: run all chunks in an Rmarkdown document using knitr::purl to get the code and sys.source to execute it.

runAllChunks <- function(rmd, envir=globalenv()){
  # as found here https://stackoverflow.com/questions/24753969
  tempR <- tempfile(tmpdir = '.', fileext = ".R")
  on.exit(unlink(tempR))
  knitr::purl(rmd, output=tempR, quiet=TRUE)
  sys.source(tempR, envir=envir)
}

For some reason, this produces an error since maybe a few weeks (not sure which new packages I installed lately...). But since there is a new comment, that I can just use knitr::knit which also executes the code, this worked as expected and now my test no longer complains.

So in the end, I don't know where the problem exactly was, but this is now working.

Answer 1

I ran into the same issue with my tests failing under devtools::check() while not failing under testthat::test()

And none of the above applied to my problem, so i decided to post my issue plus solution here as well. But first some NOTEs from my experience:

devtools::check() does - so it seems - deeper error checking then your own written tests.

Now to my code-setup. I had a function that was build to retrieve values from two different files. Those files contained named profiles with a set of values per profile. But the profiles were named differently, depending on the files:

Example files:

Content of file_one:

[default]
value_A = "foo"
value_B = "bar"
value_C = "baz"

[peter]
value_A = "oof"
value_B = "rab"
value_C = "zab"

content of file_two:

[default]
value_X = "fuzzly"
value_Z = "puzzly"

[profile peter]
value_X = "fuzzly"
value_Z = "puzzly"

As you can see, does the naming in file two follow another naming convention, when it comes to the named profiles. The profiles are written in "[]" and the default-profile is always '[default]' in both files. But as soon as it comes to named profiles, its just '[name]' in one file and then '[profile name]' in the other one.

Now i've build the function like that (simplyfied):

get_value <- function(file_content, what, profile) {
  file_content <- readr::read_lines(file)
  all_profiles_at <- grep("\\[.*\\]", file_content)
  profile_regex <- paste0("\\[",if(file_content == "file_two" && profile != "default") "profile ",profile,"\\]")
  profile_at <- grep(profile_regex, file_content)
  profile_ends_at <- if(profile_at == max(all_profiles_at)) length(file_content) else all_profiles_at[grep(paste0("^",profile_at,"$"), all_profiles_at) + 1] -1
  profile_content <- file_content[profile_at:profile_ends_at]
  whole_what <- stringr::str_replace_all(profile_content[grep(paste0("^",what,".*"), profile_content)], " ", "")
  return(stringr::str_sub(whole_what, stringr::str_length(paste0(what,"=."))))
}

With this code my tests ran smoothly and even check() found no issues.

While the whole code evolved i figured, that i should read the files content beforehand and give only the alread read_in content to the function to avoid duplication in my code. So i changed the function like so:

get_value <- function(file, what, profile) {
  is_file_two <- is_file_two(file_content)
  all_profiles_at <- grep("\\[.*\\]", file_content)
  profile_regex <- paste0("\\[",if(file_content == "file_two" && profile != "default") "profile ",profile,"\\]")
  profile_at <- grep(profile_regex, file_content)
  profile_ends_at <- if(profile_at == max(all_profiles_at)) length(file_content) else all_profiles_at[grep(paste0("^",profile_at,"$"), all_profiles_at) + 1] -1
  profile_content <- file_content[profile_at:profile_ends_at]
  whole_what <- stringr::str_replace_all(profile_content[grep(paste0("^",what,".*"), profile_content)], " ", "")
  return(stringr::str_sub(whole_what, stringr::str_length(paste0(what,"=."))))
}   

As you might notice i only changed the first line of the funciton body and left the if-condition unchanged - my mistake!

But my tests didn't throw an error, as the if-condition still worked. Even though the 'file_content == "file_two"' part now generated a logical vector and if() ... else ... normally throws a warning, when the logical has length > 1. The special construct with the && doesn't throw such an error as it returns a length(1) logical:

# with warning
if(c(FALSE, FALSE, FALSE)) "Done!" else "Not done!"
# no warning:
if(c(FALSE, FALSE, FALSE) && TRUE) "Done!" else "Not done!"

Thats why my tests with testthat::test() sill worked.

But devtools::check() saw this flaw in my code and the tests failed!

And that part of the FAILURE_REPORT showed me my errors:

[...]
    where 41: test_check("my_package_name")
    
     --- value of length: 18 type: logical ---
     [1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
    [13] FALSE FALSE FALSE FALSE FALSE FALSE
     --- function from context ---
[...]

Conclusion: testthat::test() is great! Is checks whether or not your code still runs. But devtools::check() goes far deeper - and when your tests pass with testthat::test() but fail with devtools::check() then you've probaly got some deeper bugs and flaws in your code you MUST attend to!