Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

sparkR sql() returns string

Tags:

r

parquet

sparkr

We have parquet data saved on a server and I am trying to use SparkR sql() function in the following ways

df <- sql("SELECT * FROM parquet.`<path to parquet file`")
head(df)
show(df) # returns "<SQL> SELECT * FROM parquet.`<path to parquet file`"

and

createOrReplaceTempView(df, "table")
df2 <- sql("SELECT * FROM table")
show(df2) # returns "<SQL> SELECT * FROM table"

In both cases what I get is the sql query in a string format instead of the spark dataframe. Does anyone have any idea why this happens and why I don't get the dataframe?

Thank you

like image 346
Ashkan Avatar asked Mar 03 '26 08:03

Ashkan


1 Answers

This was a very silly problem. The solution is to just use the full name of the method

SparkR::sql(...)

instead of the short name. Apparently the function sql is masked.

like image 178
Ashkan Avatar answered Mar 04 '26 21:03

Ashkan



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!