How to test graphical output of functions?

Question

I am wondering how to test functions that produce graphics. I have a simple plotting function img:

img <- function() {   plot(1:10) } 

In my package I like to create a unit test for this function using testthat. Because plot and its friends in base graphics just return NULL a simple expect_identical is not working:

library("testthat")  ## example for a successful test expect_identical(plot(1:10), img()) ## equal (as expected)  ## example for a test failure expect_identical(plot(1:10, col="red"), img()) ## DOES NOT FAIL! # (because both return NULL) 

First I thought about plotting into a file and compare the md5 checksums to ensure that the output of the functions is equal:

md5plot <- function(expr) {   file <- tempfile(fileext=".pdf")   on.exit(unlink(file))   pdf(file)   expr   dev.off()   unname(tools::md5sum(file)) }  ## example for a successful test expect_identical(md5plot(img()),                  md5plot(plot(1:10))) ## equal (as expected)  ## example for a test failure expect_identical(md5plot(img()),                  md5plot(plot(1:10, col="red"))) ## not equal (as expected) 

That works well on Linux but not on Windows. Surprisingly md5plot(plot(1:10)) results in a new md5sum at each call. Aside this problem I need to create a lot of temporary files.

Next I used recordPlot (first creating a null-device, call the plotting function and record its output). This works as expected:

recPlot <- function(expr) {   pdf(NULL)   on.exit(dev.off())   dev.control(displaylist="enable")   expr   recordPlot() }  ## example for a successful test expect_identical(recPlot(plot(1:10)),                  recPlot(img())) ## equal (as expected)  ## example for a test failure expect_identical(recPlot(plot(1:10, col="red")),                  recPlot(img())) ## not equal (as expected) 

Does anybody know a better way to test the graphical output of functions?

EDIT: regarding the points @josilber asks in his comments.

While the recordPlot approach works well you have to rewrite the whole plotting function in the unit test. That becomes complicated for complex plotting functions. It would be nice to have an approach that allows to store a file (*.RData or *.pdf, ...) which contains an image against you could compare in future tests. The md5sum approach isn't working because the md5sums differ on different platforms. Via recordPlot you could create an *.RData file but you could not rely on its format (from the recordPlot manual page):

The format of recorded plots may change between R versions. Recorded plots can not be used as a permanent storage format for R plots.

Maybe it would be possible to store an image file (*.png, *.bmp, etc), import it and compare it pixel by pixel ...

EDIT2: The following code illustrate the desired reference file approach using svg as output. First the needed helper functions:

## plot to svg and return file contant as character plot_image <- function(expr) {   file <- tempfile(fileext=".svg")   on.exit(unlink(file))   svg(file)   expr   dev.off()   readLines(file) }  ## the IDs differ at each `svg` call, that's why we simple remove them ignore_svg_id <- function(lines) {   gsub(pattern = "(xlink:href|id)=\"#?([a-z0-9]+)-?(?<![0-9])[0-9]+\"",        replacement = "\\1=\"\\2\"", x = lines, perl = TRUE) }  ## compare svg character vs reference expect_image_equal <- function(object, expected, ...) {   stopifnot(is.character(expected) && file.exists(expected))   expect_equal(ignore_svg_id(plot_image(object)),                ignore_svg_id(readLines(expected)), ...) }  ## create reference image create_reference_image <- function(expr, file) {   svg(file)   expr   dev.off() } 

A test would be:

create_reference_image(img(), "reference.svg")  ## create tests library("testthat")  expect_image_equal(img(), "reference.svg") ## equal (as expected) expect_image_equal(plot(1:10, col="red"), "reference.svg") ## not equal (as expected) 

Sadly this is not working across different platforms. The order (and the names) of the svg elements completely differs on Linux and Windows.

Similar problems exist for png, jpeg and recordPlot. The resulting files differ on all platforms.

Currently the only working solution is the recPlot approach above. But therefore I need to rewrite the whole plotting functions in my unit tests.

---

P.S.: I am completley confused about the different md5sums on Windows. It seems they depending on the creation time of the temporary files:

# on Windows table(sapply(1:100, function(x)md5plot(plot(1:10)))) #4693c8bcf6b6cb78ce1fc7ca41831353 51e8845fead596c86a3f0ca36495eacb #                              40                               60 

Answer 1

Mango Solutions have published an open source package, visualTest, that does fuzzy matching of plots, to address this use case.

The package is on github, so install using:

devtools::install_github("MangoTheCat/visualTest") library(visualTest) 

Then use the function getFingerprint() to extract a finger print for each plot, and compare using the function isSimilar(), specifying a suitable threshold.

First, create some plots on file:

png(filename = "test1.png") img() dev.off()  png(filename = "test2.png") plot(1:11, col="red") dev.off() 

The finger print is a numeric vector:

> getFingerprint(file = "test1.png")  [1]  4  7  4  4 10  4  7  7  4  7  7  4  7  4  5  9  4  7  7  5  6  7  4  7  4  4 10 [28]  4  7  7  4  7  7  4  7  4  3  7  4  4  3  4  4  5  5  4  7  4  7  4  7  7  7  4 [55]  7  7  4  7  4  7  5  6  7  7  4  8  6  4  7  4  7  4  7  7  7  4  4 10  4  7  4  > getFingerprint(file = "test2.png")  [1]  7  7  4  4 17  4  7  4  7  4  7  7  4  5  9  4  7  7  5  6  7  4  7  7 11  4  7 [28]  7  5  6  7  4  7  4 14  4  3  4  7 11  7  4  7  5  6  7  7  4  7 11  7  4  7  5 [55]  6  7  7  4  8  6  4  7  7  4  4  7  7  4 10 11  4  7  7 

Compare using isSimilar():

> isSimilar(file = "test2.png", +           fingerprint = getFingerprint(file = "test1.png"), +           threshold = 0.1 + ) [1] FALSE 

---

You can read more about the package at http://www.mango-solutions.com/wp/products-services/r-services/r-packages/visualtest/

Answer 2

It's worth noting that the vdiffr package also supports comparing plots. A nice feature is that it integrates with the testthat package -- it's actually used for testing in ggplot2 -- and it has an add-in for RStudio to help manage your testsuite.