Convert an ff object to a data.frame

Question

I am working with big matrix and the ff package. I am loading an ff object and I want to use it to calculate a crps (a score).

For example, I have a ff_matrix (called Mat with 25 rows and 7303 columns) which is a precipitation forecast (7303 represents the number of days (about 20 years) and 25 are the 25 precipitation simulations for one day). I also have a ff_array with the observations for these 20 years (called Obs and with 7303 values).

With the package ensembleBMA I want to calculate the CRPS. I need to put my ff_matrix and my ff_array in an "ensembleBMA" object (in fact this is a data.frame).

For this code:

ensembleBMA(Mat,Obs)

I have this error:

Error in as.data.frame.default(x[[i]], optional = TRUE, stringsAsFactors = stringsAsFactors) : cannot coerce class 'c("ff_matrix", "ff_array", "ff")' into a data.frame

I tried different options such as:

as.data.frame(Mat)
as.matrix(Mat)
transform.ffdf(as.ffdf(Mat))

I always have these errors:

Error in as.data.frame.default(Mat_Ptot_212_1) : cannot automatically convert class  'c("ff_matrix", "ff_array", "ff")' into a data frame (data.frame)

or

opening ff /tmp/RtmpWrlY4n/clone9d3376b435.ff Error in ff(initdata = initdata, length = length, levels = levels, ordered = ordered,  : write error

Does someone has an idea?

Answer 1

One way us to first convert your ff_array to an array and convert that to a data.frame:

Mat <- ff(1, vmode="double", dim=c(25, 7303))
as.data.frame(Mat[,])

or first convert your ff_array to an ffdf and convert that to an data.frame:

 as.ffdf(Mat)[,]

or

as.data.frame(as.ffdf(Mat))

The last two solutions seem to be much slower than the first. This has probably to do with the large number of columns which slows down as.ffdf which has to create 7303 files.

There does not seem to be a as.data.frame.ff_array.