Why does data.table get copied when adding a new column?

Question

When a new column is added to a data.table that is loaded from disk, it get copied.

library('data.table')
dt <- data.table(a=1,b=2)
save.image("test.RData")
load("test.RData")
dt
$   a b
$1: 1 2

class(dt)
$[1] "data.table" "data.frame"

address(dt)
$[1] "00000000046F1F38"  

dt[, b := NULL]
address(dt)
$[1] "00000000046F1F38"
dt[, c := 2]
address(dt)
$[1] "000000000D815618"

Is this a bug or am I doing something wrong? I am using 1.9.6 of the data.table package.

Answer 1

data.table avoids copies when adding columns by over-allocating pointer slots for the list of column vectors when the data.table is created. When you load the data.table like this, over-allocation has not happend and is done once you add a column. This makes a copy necessary.

library('data.table')
dt <- data.table(a=1,b=2)
save.image("test.RData")
load("test.RData")

truelength(dt)
#[1] 0

dt[, b := NULL]
truelength(dt)
#[1] 0

dt[, c := 2]
truelength(dt)
#[1] 101

To quote help("truelength"):

For tables loaded from disk however, truelength is 0 in R 2.14.0 and random in R <= 2.13.2; i.e., in both cases perhaps unexpected. data.table detects this state and over-allocates the loaded data.table when the next column addition or deletion occurs. All other operations on data.table (such as fast grouping and joins) do not need truelength.

It seems that the documentation is slightly out of date since the copy doesn't happen during deletion of a column.

Note that a copy also happens if you add more columns than have been over-allocated during "normal" creation of a data.table.