Fast grouping by list column subsets in data.table

Question

I am working with a large (millions of rows) data table with a list column containing deeply nested lists, which do not have uniform structure, size or order of elements (list(x=1,y=2) and list(y=2,x=1) may both be present and should be treated as identical). I need to repeatedly perform arbitrary groupings that include some columns from the data table as well as a subset of the data in the list column. Not all rows have values that will match the subset.

The approach I've come up with feels overly complicated. Here are the key points:

• Identifying values in a nested list structure. My approach is to use ul <- unlist(list_col), which "flattens" nested data structures and builds hierarchical names for direct access to each element, e.g., address.country.code.

• Ensuring that permutations of the same unlisted data are considered equal from a grouping standpoint. My approach is to order the unlisted vectors by the names of their values via ul[order(names(ul))] and assign the result as a new character vector column by reference.

• Performing grouping on subsets of the flattened values. I was not able to get by= to work in any way with a column whose values are lists or vectors. Therefore, I had to find a way to map unique character vectors to simple values. I did this with digest.

Here are the two workhorse functions:

# Flatten list column in a data.table
flatten_list_col <- function(dt, col_name, flattened_col_name='props') {

  flatten_props <- function(d) {
    if (length(d) > 0) {
      ul <- unlist(d)
      names <- names(ul)
      if (length(names) > 0) {
        ul[order(names)]          
      } else {
        NA
      }
    } else {
      NA
    }
  }

  flattened <- lapply(dt[[col_name]], flatten_props)
  dt[, as.character(flattened_col_name) := list(flattened), with=F]
}

# Group by properties in a flattened list column
group_props <- function(prop_group, prop_col_name='props') {
  substitute({
    l <- lapply(eval(as.name(prop_col_name)), function(x) x[names(x) %in% prop_group])
    as.character(lapply(l, digest))
  }, list(prop_group=prop_group, prop_col_name=prop_col_name))
}

Here is a reproducible example:

library(data.table)

dt <- data.table(
  id=c(1,1,1,2,2,2), 
  count=c(1,1,2,2,3,3), 
  d=list(
    list(x=1, y=2), 
    list(y=2, x=1), 
    list(x=1, y=2, z=3),
    list(y=5, abc=list(a=1, b=2, c=3)),
    NA,
    NULL    
    )
)

flatten_list_col(dt, 'd')
dt[, list(total=sum(count)), by=list(id, eval(group_props(c('x', 'y'))))]

The output is:

> flatten_list_col(dt, 'd')
   id count      d   props
1:  1     1 <list>     1,2
2:  1     1 <list>     1,2
3:  1     2 <list>   1,2,3
4:  2     2 <list> 1,2,3,5
5:  2     3     NA      NA
6:  2     3             NA

> dt[, list(total=sum(count)), by=list(id, eval(group_props(c('x', 'y'))))]
   id                      group_props total
1:  1 325c6bbb2c33456d0301cf3909dd1572     4
2:  2 7aa1e567cd0d6920848d331d3e49fb7e     2
3:  2 ee7aa3b9ffe6bffdee83b6ecda90faac     6

This approach works but is pretty inefficient because of the need to flatten & order the lists and because of the need to calculate digests. I'm wondering about the following:

1. Can this be done without having to create a flattened column by instead retrieving values directly from the list column? This will probably require specifying selected properties as expressions as opposed to simple names.

2. Is there a way to get around the need for digest?

Answer 1

There are a number of issues here. The most important (and one you haven't come to yet due to others) is that that you are assigning by reference but trying to replace with more values than you have space to do so by reference.

Take this very simple example

DT <- data.table(x=1, y = list(1:5))
DT[,new := unlist(y)]
Warning message:
In `[.data.table`(DT, , `:=`(new, unlist(y))) :
  Supplied 5 items to be assigned to 1 items of column 'new' (4 unused)

You will lose all but the firstnrow(DT) items in the newly created list. They wont correspond to the rows of the data.table

Therefore you will have to create a new data.table that will be large enough for you to explode these list variables. This won't be possible by reference.

 newby <- dt[,list(x, props = as.character(unlist(data))), by = list(newby = seq_len(nrow(dt)))][,newby:=NULL]
newby


   x props
 1: 1     1
 2: 1     2
 3: 1     2
 4: 1     1
 5: 1    10
 6: 2     1
 7: 2     2
 8: 2     3
 9: 2     5
10: 2     1
11: 2     2
12: 2     3
13: 3    NA
14: 3    NA

Note that as.character is required to ensure that all values are the same type, and a type that won't lose data in the conversion. At the momemnt you have a logical NA value amongst lists of numeric / integer data.

---

Another edit to force all components to be character (even the NA). props is now a list with 1 character vector for each row.

flatten_props <- function(data) { if (is.list(data)){ ul <- unlist(data) if (length(ul) > 1) { ul <- ul[order(names(ul))] } as.character(ul) } else { as.character(unlist(data))}}

dt[, props := lapply(data, flatten_props)]
dt
   x   data   props
1: 1 <list>     1,2
2: 1 <list>  10,1,2
3: 2 <list>   1,2,3
4: 2 <list> 1,2,3,5
5: 3     NA      NA
6: 3   

dt[,lapply(props,class)]
          V1        V2        V3        V4        V5        V6
1: character character character character character character