Join inherited scope with by in data.table

Question

I'm on data.table 1.9.3, and maybe I'm wrong, but I don't recall the following to be expected previously.

I build 2 data.tables, dta and dtb

> dta
   idx vala fdx
1:   1    2   a
2:   2    4   a
3:   3    6   b

> dtb
   idx valb
1:   1    3
2:   4    6

> dput(x = dta)
structure(list(idx = c(1, 2, 3), vala = c(2, 4, 6), fdx = c("a",
"a", "b")), .Names = c("idx", "vala", "fdx"), row.names = c(NA,
-3L), class = c("data.table", "data.frame"), .internal.selfref =
<pointer: 0x0000000000110788>, sorted = "idx")

> dput(x = dtb)
structure(list(idx = c(1, 4), valb = c(3, 6)), .Names = c("idx",
"valb"), row.names = c(NA, -2L), class = c("data.table", "data.frame"
), .internal.selfref = <pointer: 0x0000000000110788>, sorted = "idx")

The key is idx in both cases.

The following works, of course

> dta[dtb, sum(valb)]
[1] 9

However this doesn't

> dta[dtb, sum(valb), by = fdx]
Error in `[.data.table`(dta, dtb, sum(valb), by = fdx) :
  object 'valb' not found

But this does

> dta[dtb][, sum(valb), by = fdx]
   fdx V1
1:   a  3
2:  NA  6

If we see the intermediate step

> dta[dtb]
   idx vala fdx valb
1:   1    2   a    3
2:   4   NA  NA    6

I would have expected

dta[dtb, sum(valb), by = fdx] == dta[dtb][, sum(valb), by = fdx]

Where have I gone wrong?

Answer 1

Just a guess

library(data.table)

dta <- data.frame(idx=c(1,2,3), 
                  vala=c(2,4,6),
                  fdx=c('a','a','b'))
dta <- data.table(dta)

dtb <- data.frame(idx=c(1,4),
                  valb=c(3,6))
dtb <- data.table(dtb)

setkey(dta,idx)
setkey(dtb,idx)

So when you call

dta[dtb, sum(valb)]

it's kind of like calling

tmp <- dta[dtb]
attach(tmp)
sum(valb)
detach(tmp)

However, if you call

dta[dtb, sum(valb), by=fdx]

then it's kind of like calling

tmp <- dta[dtb]
# attach(tmp) : attach doesn't happen
sum(valb)
# detach(tmp) : detach doesn't happen

The function doesn't know what to do with the additional arguments. For instance, this would also throw an error:

dta[dtb, class(fdx), sum(valb)]

However, this works

dta[dtb][, sum(valb), by=fdx]

which is kind of like

tmp <- dta[dtb]
tmp[, sum(valb), by=fdx]

Like I said, this is just a guess as to why the function may not be working as expected.