Menu
We have tried wrapping the column name with brackets [column name], single & double quotes, and backticks, none of them works.
Does Spark SQL support columns whose name contains spaces?
480
To select a column name with spaces, use the back tick symbol with column name. The symbol is ( ` `). Back tick is displayed in the keyboard below the tilde operator ( ~).
Column names can contain any valid characters (for example, spaces).
To Remove both leading and trailing space of the column in pyspark we use trim() function. trim() Function takes column name and trims both left and right white space from that column.
In Spark SQL, select () function is used to select one or multiple columns, nested columns, column by index, all columns, from the list, by regular expression from a DataFrame. select () is a transformation function in Spark and returns a new DataFrame with the selected columns. You can also alias column names while selecting.
To select a column name with spaces, use the back tick symbol with column name. The symbol is ( ` `). Back tick is displayed in the keyboard below the tilde operator ( ~).
The syntax to get column name with space is as follows −. SELECT `column_name` from yourTableName; Now I will apply the above syntax to get the result for my column. The query is as follows −. mysql> SELECT `Student Name` from SpaceColumn; The following is the output −.
In order to access PySpark/Spark DataFrame Column Name with a dot from wihtColumn () & select (), you just need to enclose the column name with backticks (`) Using Column Name with Dot on select (). Using Column Name with Dot on withColumn ()
Backticks seem to work just fine:
scala> val df = sc.parallelize(Seq(("a", 1))).toDF("foo bar", "x") df: org.apache.spark.sql.DataFrame = [foo bar: string, x: int] scala> df.registerTempTable("df") scala> sqlContext.sql("""SELECT `foo bar` FROM df""").show foo bar a Same as DataFrame API:
scala> df.select($"foo bar").show foo bar a So it looks like it is supported, although I doubt it is recommended.
115
Instead of using Brackets like in T-SQL [column name] Use backticks to wrap the column name `column name`. This is when you run SQL. You also use Backticks in spark SQL to wrap the column name but use triple quotes as answered by zero323.
31
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
