Modifying package gbm of R

Question

We are trying to experiment using the gbm package on a quite large dataset (~140 million rows) and we have ran into a problem with the memory requirements of R.

We have tried combining the packages 'gbm' and 'bigmemory' with no success and our next thought was to modify the C++ source code to draw data from a local database where we have stored our dataset.

So, we were wondering if there is a more appropriate or well-known practice in order to change the allocation inside the C++ code of gbm. Has anyone tried something similar?

Answer 1

I’m not familiar with the gbm package, but if it works on data frames or vectors of some kind you could use the ff package.

Quote: The ff package provides data structures that are stored on disk but behave (almost) as if they were in RAM by transparently mapping only a section (pagesize) in main memory...