Is there a way to load .RData or .model files in sparkR (using databricks)?

Question

I'm using sparkR inside databricks environment and I'm trying to load an XGB model that has been created using RStudio.

I saved the model both in .RData and .model (using xgb.save) types (just to increase my chances of making this work) and then uploaded both files to a folder inside DBFS (databricks).

Now, using databricks, I wanted to load the model so I can apply it to new data, but I always got the same error message. Here's the code and the error:

%r
# Trying to load the .RData file
load(file='/FileStore/tables/folder/xgb_model.RData')

Error in readChar(con, 5L, useBytes = TRUE) : cannot open the connection Error in readChar(con, 5L, useBytes = TRUE) : cannot open the connection Error in readChar(con, 5L, useBytes = TRUE) : cannot open the connection

%r
# Trying to load the .model file
xgb.load("/FileStore/tables/folder/xgb_model.model")

Error in xgb.Booster.handle(modelfile = modelfile) : Error in xgb.Booster.handle(modelfile = modelfile) : [23:34:59] amalgamation/../dmlc-core/src/io/local_filesys.cc:209: Check failed: allow_null: LocalFileSystem::Open "/FileStore/tables/folder/xgb_model.model": No such file or directory

Is that even possible?

Answer 1

The FileStore is a special folder that is located on DBFS, so to access this data, R code needs to understand DBFS. But in your case, it can only work with the local files, that why it's failing. To make it working with files in FileStore as local files, you need to prepend the /dbfs/ to the file names, like, /dbfs/FileStore/tables/folder/xgb_model.model (please note that it may not work on the community edition with 7.x runtime)