Why `missing` and default arguments are not working in functions called by `lapply`?

Question

I'm astonished that missing seems not working in a function called by lapply. Assume I have the following functions:

.add <- function(x, arg, ...) {
  if (missing(arg)) {
    arg <- 1
  }
  print(match.call())
  return(x + arg)
}

wrapper <- function(l, arg, ...) {
  return(lapply(l, .add, arg=arg, ...))
}

Setting arg explicit works like excepted:

wrapper(list(x=1:10, y=1:10), arg=1)
#FUN(x = X[[1L]], arg = ..1)
#FUN(x = X[[2L]], arg = ..1)
#$x
# [1]  2  3  4  5  6  7  8  9 10 11
#
#$y
# [1]  2  3  4  5  6  7  8  9 10 11

Without arg I would expect the same output but it fails:

wrapper(list(x=1:10, y=1:10))
#FUN(x = X[[1L]], arg = ..1)
# Error in FUN(X[[1L]], ...) : argument "arg" is missing, with no default

missing works in nested wrapper functions where no lapply is used. Why it seems to have no effect in functions called by lapply?

EDIT: Default arguments also don't work:

.add <- function(x, arg=5, ...) {
  if (missing(arg)) {
    arg <- 1
  }
  print(match.call())
  return(x + arg)
}

wrapper(list(x=1:10, y=1:10))
#FUN(x = X[[1L]], arg = ..1)
# Error in FUN(X[[1L]], ...) : argument "arg" is missing, with no default

It seems that arg is neither missing nor accessible. What happens here?

(I know that I could circumvent this by setting arg=NULL in wrapper and if (is.null(arg)) in .add or something else. .add is an internal function which determines arg by its own based on the input (e.g. arg=mean(x)) and I want arg in the wrapper to document the argument arg for the user and to allow the user to overwrite the default behavior. And most important: I want to understand why this is not working!)

EDIT2: Finally this behaviour is fixed. It was a bug in R < 3.2.0, see PR#15707.

Answer 1

First, I'll mention that I believe the idiomatic way of doing this is by constructing a call and then evaluating it. See write.csv for an example. I believe this code will do what you want, using that method.

wrapper <- function(X, arg, ...) {
  force(X) # optional; if X is missing, the error message will be more informative
  Call <- match.call(expand.dots=TRUE)
  Call[[1L]] <- as.name("lapply")
  Call$FUN <- as.name(".add")
  eval.parent(Call)
}

Ok, now here's an attempt to explain the issues you discovered. I stand ready to be corrected as well, but hopefully this will at least help clarify the issues, just like @idfah's answer did.

First, I'll tackle the "defaults" issue, as I think it's more straightforward. This one I think can be made simpler, as in the following two functions, where the second (f2) simply calls the first (f1). What we see is that the default argument in f1 gets overridden by the promise to x in f2, and when that promise is evaluated, it is missing. Moral of this story (I think); defaults must be set again in your calling function, if that variable is included in the call.

f1 <- function(x=1) {print(match.call()); x}
f2 <- function(x) {f1(x=x)}
f1()
## f1()
## [1] 1
f2()
## f1(x = x)
## Error in f1(x = x) : argument "x" is missing, with no default

Now on to the missing in lapply issue. Here I basically have sgibb's code, but have added a message about whether or not arg is considered missing. We have what seems to be a curious contradiction; the message tells us that arg is NOT missing, but when the function tries to access it, we get an error message telling us that arg IS missing.

.add <- function(x, arg) {
  print(match.call())
  if(missing(arg)) {
    message("arg is missing in .add")
    x
  } else {
    message("arg is not missing")
    x + arg
  }
}
wrapper <- function(l, arg) {lapply(l, .add, arg=arg)}
wrapper(1)
## FUN(x = 1[[1L]], arg = ..1)
## arg is not missing
## Error in FUN(1[[1L]], ...) : argument "arg" is missing, with no default

What I think is happening is the lapply is putting the promise to arg in ..1, so it doesn't look missing, but when it tries to evaluate it, it finds that it is missing. Moral of this story (I think); don't try to propagate missings through lapply.

UPDATE: More precisely, it's something with how dot expansion works. Consider this version of lapply (which doesn't actually work on a list, but otherwise has the same code style); this shows we get the same behavior.

apply3 <- function(X, FUN, ...) { 
  print(match.call())
  FUN(X, ...)
}
wrapper3 <- function(l, arg) {apply3(l, .add, arg=arg)}
wrapper3(1)
## apply3(X = l, FUN = .add, arg = arg)
## FUN(x = X, arg = ..1)
## arg is not missing
## Error in FUN(X, ...) : argument "arg" is missing, with no default

But when we substitute the dots with a variable name, it works as expected.

apply4 <- function(X, FUN, hm) { 
  print(match.call())
  FUN(X, hm)
}
wrapper4 <- function(l, arg) {apply4(l, .add, hm=arg)}
wrapper4(1)
## apply4(X = l, FUN = .add, hm = arg)
## FUN(x = X, arg = hm)
## arg is missing in .add
## [1] 1

And one more example; if I use dots, but do the expansion myself, by calling ..1 directly, it also works! This is curious as the matched call is the same as the version that doesn't work.

apply3b <- function(X, FUN, ...) { 
  print(match.call())
  FUN(X, ..1)
}
wrapper3b <- function(l, arg) {apply3b(l, .add, arg=arg)}
wrapper3b(1)
## apply3b(X = l, FUN = .add, arg = arg)
## FUN(x = X, arg = ..1)
## arg is missing in .add
## [1] 1

Answer 2

There is no missing in your wrapper, so it bombs there. In this case, you don't need it since you are using variadic arguments anyway. Try this:

.add <- function(x, arg, ...) {
    if (missing(arg)) 
      arg <- 1
    print(match.call())
  return(x + arg)
}

wrapper <- function(l, ...) 
  return(lapply(l, .add, ...))

If the wrapper needs to know arg, then you need an missing there:

.add <- function(x, arg, ...) {
  print(match.call())
  return(x + arg)
}

wrapper <- function(l, ...) {
  if (missing(arg)) 
    arg <- 1
  return(lapply(l, .add, arg=arg, ...))
}

I stand corrected

The following example allows the missing to be at the bottom of the call stack, presumably because of lazy evaluation. I am unsure then why your example does not work... curious.

wrapper.c <- function(l, arg)
{
  if (missing(arg))
    arg <- 1
  print("I'm in c")
  arg
}

wrapper.b <- function(l, arg)
{
  print("I'm in b")
  wrapper.c(l, arg)
}

wrapper.a <- function(l, arg)
  wrapper.b(l, arg)

> wrapper.a(1)
[1] "I'm in b"
[1] "I'm in c"
[1] 1