How do I read a Parquet in R and convert it to an R DataFrame?

Question

I'd like to process Apache Parquet files (in my case, generated in Spark) in the R programming language.

Is an R reader available? Or is work being done on one?

If not, what would be the most expedient way to get there? Note: There are Java and C++ bindings: https://github.com/apache/parquet-mr

Answer 1

If you're using Spark then this is now relatively simple with the release of Spark 1.4 see sample code below that uses the SparkR package that is now part of the Apache Spark core framework.

# install the SparkR package devtools::install_github('apache/spark', ref='master', subdir='R/pkg')  # load the SparkR package library('SparkR')  # initialize sparkContext which starts a new Spark session sc <- sparkR.init(master="local")  # initialize sqlContext sq <- sparkRSQL.init(sc)  # load parquet file into a Spark data frame and coerce into R data frame df <- collect(parquetFile(sq, "/path/to/filename"))  # terminate Spark session sparkR.stop() 

An expanded example is shown @ https://gist.github.com/andyjudson/6aeff07bbe7e65edc665

I'm not aware of any other package that you could use if you weren't using Spark.