Skip tests on CRAN, but run locally

Question

It there an easy way to skip the execution of some tests in a package if the package is tested by CRAN? The background is, I like to have a lot of tests and in sum they are time consuming (not good for CRAN).

I know there is testthat::skip_on_cran() but I do not want to use package testthat to avoid another dependency. I am looking for an el-cheapo way to mimic testthat::skip_on_cran.

Ideally, I would like to have a testfile in directory pkg/tests that calls the tests (testfiles) and distuingishes if we are on cran or not:

if (!on_cran) {
 ## these tests are run only locally/when not on CRAN
 # run test 1 (e.g. source a file with a test)
 # run test 2
}
# run test 3 - always

Answer 1

Yes! You can handle this programmatically and automatically. Let me detail two ways I've set up:

Implicitly via version numbers: This is the approach taken by Rcpp for many years now, and it is entirely generic and not dependent on any other package. Our tests start from a file in tests/ and then hand over to RUnit, but that last part is an implementation detail.

In the main file tests/doRUnit.R we do this:

## force tests to be executed if in dev release which we define as
## having a sub-release, eg 0.9.15.5 is one whereas 0.9.16 is not
if (length(strsplit(packageDescription("Rcpp")$Version, "\\.")[[1]]) > 3) { 
    Sys.setenv("RunAllRcppTests"="yes")
}

In essence, we test if the version is of the form a.b.c.d -- and if so conclude that it is a development version. This implies "run all tests". Whereas a release version of the form a.b.c would go to CRAN, and not run these tests as they would exceed their time limit.

In each of the actual unit test files, we can then decide if we want to honour the variable and skip the test if set, or execute anyway:

.runThisTest <- Sys.getenv("RunAllRcppTests") == "yes"

if (.runThisTest) {

   ## code here that contains the tests

}

This mechanism is fully automatic, and does not depend on the user. (In the actual package version there is another if () test wrapped in there which allows us to suppress the tests, but that is a detail we do not need here).

I still like this approach a lot.

Explicitly via resource files Another package a few of us work on (a lot lately) requires a particular backend to be available. So in the Rblpapi package we tests for presence of a file which my coauthors and I each have below our $HOME directory in order to set up credentials and connection details. If the file is missing --- as e.g. on Travis CI, or CRAN, or for other users, the tests are skipped.

We chose to use the resource file as R file; it sources it if found and thereby sets values for options(). That way we can control directly whether to launch tests or not.

## We need to source an extra parameter file to support a Bloomberg connection
## For live sessions, we use ~/.Rprofile but that file is not read by R CMD check
## The file basically just calls options() and sets options as needed for blpHost,
## blpPort, blpAutoConnect (to ensure blpConnect() is called on package load) and,
## as tested for below, blpUnitTests.
connectionParameterFile <- "~/.R/rblpapiOptions.R"
if (file.exists(connectionParameterFile)) source(connectionParameterFile)

## if an option is set, we run tests. otherwise we don't.
## recall that we DO need a working Bloomberg connection...
if (getOption("blpUnitTests", FALSE)) {
  
    ## ... more stuff here which sets things up

}

Similarly to the first use case we can now set more variables which are later tested.

Explicitly via Travis CI Another option we use in rfoaas is to set the environment variable governing this in the Travis CI file:

env:
  global:
    - RunFOAASTests=yes

which the tests script then picks up:

## Use the Travis / GitHub integrations as we set this
## environment variable to "yes" in .travis.yml
##
## Set this variable manually if you want to run the tests
##
if (Sys.getenv("RunFOAASTests=yes") == "yes") runTests <- TRUE

In that case I also set the toggle based on my userid as I am pretty much the sole contributor to that project:

## Also run the tests when building on Dirk's box, even whem
## the environment variable is not set
if (isTRUE(unname(Sys.info()["user"])=="edd")) runTests <- TRUE

Explicitly via another variable You can of course also rely on another variable you use across all your packages. I find that to be a bad idea. If you set this in your shell, work on package A and set it to suppress tests but then switch to package B --- you will likely forget to unset the variable and then fail to test. I like this approach the least and do not use it.

Answer 2

Use an environment variable, like testthat does:

skip_on_cran <- function() {
  if (identical(Sys.getenv("NOT_CRAN"), "true")) {
    return(invisible(TRUE))
  }

  skip("On CRAN")
}