Menu
I have imported a json file in Spark and convertd it into a table as
myDF.registerTempTable("myDF") I then want to run SQL queries on this resulting table
val newTable = sqlContext.sql("select column-1 from myDF") However this gives me an error because of the hypen in the name of the column column-1. How do I resolve this is Spark SQL?
311
Use ` to escape special characters (e.g., ` ).
Quotation Mark " The SQL:1999 standard specifies that double quote (") (QUOTATION MARK) is used to delimit identifiers. Oracle, PostgreSQL, MySQL, MSSQL and SQlite all support " as the identifier delimiter.
By using regexp_replace() Spark function you can replace a column's string value with another string/substring. regexp_replace() uses Java regex for matching, if the regex does not match it returns an empty string. The below example replaces the street name Rd value with Road string on address column.
generally when u deal with apostrophe u replace the the single quote(') with ('').
Backticks (`) appear to work, so
val newTable = sqlContext.sql("select `column-1` from myDF") should do the trick, at least in Spark v1.3.x.
99
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
