I'm writing a R package (delftfews
) here at office. we are using svUnit
for unit testing.
our process for describing new functionality: we define new unit tests, initially marked as DEACTIVATED
; one block of tests at a time we activate them and implement the function described by the tests. almost all the time we have a small amount of DEACTIVATED tests, relative to functions that might be dropped or will be implemented.
my problem/question is: can I alter the doSvUnit.R so that R CMD check pkg
emits a NOTE (i.e. a custom message "NOTE" instead of "OK") in case there are DEACTIVATED tests?
as of now, we see only that the active tests don't give error:
. . * checking for unstated dependencies in tests ... OK * checking tests ... Running ‘doSvUnit.R’ OK * checking PDF version of manual ... OK
which is all right if all tests succeed, but less all right if there are skipped tests and definitely wrong if there are failing tests. In this case, I'd actually like to see a NOTE or a WARNING like the following:
. . * checking for unstated dependencies in tests ... OK * checking tests ... Running ‘doSvUnit.R’ NOTE 6 test(s) were skipped. WARNING 1 test(s) are failing. * checking PDF version of manual ... OK
As of now, we have to open the doSvUnit.Rout
to check the real test results.
I contacted two of the maintainers at r-forge and CRAN and they pointed me to the sources of R, in particular the testing.R
script.
if I understand it correctly, to answer this question we need patching the tools
package:
system
call,so I opened a change request on R, proposing something like bit-coding the return status, bit-0 for ERROR (as it is now), bit-1 for WARNING, bit-2 for NOTE.
with my modification, it would be easy producing this output:
. . * checking for unstated dependencies in tests ... OK * checking tests ... Running ‘doSvUnit.R’ NOTE - please check doSvUnit.Rout. WARNING - please check doSvUnit.Rout. * checking PDF version of manual ... OK
Brian Ripley replied "There are however several packages with properly written unit tests that do signal as required. Please do take this discussion elsewhere: R-bugs is not the place to ask questions." and closed the change request.
anybody has hints?
You should be able to alter the doSvUnit.R script to at least emit warnings and errors as you describe. What you want to do is to run the tests and then inspect the return value of the test runner and have R code that calls, warning() or stop().
For an example of how this was done using RUnit, take a look at the codetoolsBioC package in the Bioconductor repository. The relevant code is in inst/templates and copied below:
.test <- function(dir, pattern = ".*_test\\.R$") { .failure_details <- function(result) { res <- result[[1L]] if (res$nFail > 0 || res$nErr > 0) { Filter(function(x) length(x) > 0, lapply(res$sourceFileResults, function(fileRes) { names(Filter(function(x) x$kind != "success", fileRes)) })) } else list() } if (missing(dir)) { dir <- system.file("unitTests", package="@PKG@") if (!nzchar(dir)) { dir <- system.file("UnitTests", package="@PKG@") if (!nzchar(dir)) stop("unable to find unit tests, no 'unitTests' dir") } } ## Run unit tests from the directory containing the test files. ## This allows tests to refer to data files with relative paths cwd <- getwd() on.exit(setwd(cwd)) setwd(dir) require("RUnit", quietly=TRUE) || stop("RUnit package not found") RUnit_opts <- getOption("RUnit", list()) RUnit_opts$verbose <- 0L RUnit_opts$silent <- TRUE RUnit_opts$verbose_fail_msg <- TRUE options(RUnit = RUnit_opts) suite <- defineTestSuite(name="@PKG@ RUnit Tests", dirs=getwd(), testFileRegexp=pattern, rngKind="default", rngNormalKind="default") result <- runTestSuite(suite) cat("\n\n") printTextProtocol(result, showDetails=FALSE) if (length(details <- .failure_details(result)) >0) { cat("\nTest files with failing tests\n") for (i in seq_along(details)) { cat("\n ", basename(names(details)[[i]]), "\n") for (j in seq_along(details[[i]])) { cat(" ", details[[i]][[j]], "\n") } } cat("\n\n") stop("unit tests failed for package @PKG@") } result }
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With