Menu
How we can pass a column name and operator name dynamically to the SQL query with Spark in Scala?
I tried (unsuccessfully) the following:
spark.sql("set key_tbl=mytable")
spark.sql("select count(1) from ${key_tbl}").collect()
708
Spark SQL lets you query structured data inside Spark programs, using either SQL or a familiar DataFrame API. Usable in Java, Scala, Python and R. Apply functions to results of SQL queries.
Extrapolating the average I/O rate across the duration of the tests (Big SQL is 3.2x faster than Spark SQL), then Spark SQL actually reads almost 12x more data than Big SQL, and writes 30x more data.
Spark Streaming It allows us to build a scalable, high-throughput, and fault-tolerant streaming application of live data streams. Spark Streaming supports the processing of real-time data from various input sources and storing the processed data to various output sinks.
Test results: RDD's outperformed DataFrames and SparkSQL for certain types of data processing. DataFrames and SparkSQL performed almost about the same, although with analysis involving aggregation and sorting SparkSQL had a slight advantage.
You can pass it as parameter as shown below
val param = "tableName"
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
sqlContext.sql(s"""SELECT * FROM param=$param""")
can check this link for more details https://forums.databricks.com/questions/115/how-do-i-pass-parameters-to-my-sql-statements.html
113
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
