Menu
val columnName=Seq("col1","col2",....."coln"); Is there a way to do dataframe.select operation to get dataframe containing only the column names specified . I know I can do dataframe.select("col1","col2"...) but the columnNameis generated at runtime. I could do dataframe.select() repeatedly for each column name in a loop.Will it have any performance overheads?. Is there any other simpler way to accomplish this?
820
In Spark, you can use either sort() or orderBy() function of DataFrame/Dataset to sort by ascending or descending order based on single or multiple columns, you can also do sorting using Spark SQL sorting functions, In this article, I will explain all these different ways using Scala examples.
You can get the all columns of a Spark DataFrame by using df. columns , it returns an array of column names as Array[Stirng] .
First, Create a Spark Dataframe. 1. Select Single & Multiple Columns You can select the single or multiple columns of the Spark DataFrame by passing the column names you wanted to select to the select () function.
This example is also available at Spark Scala GitHub Project for reference. 10. Conclusion In this article, you have learned select () is a transformation function of the DataFrame and is used to select one or more columns, you have also learned how to select nested elements from the DataFrame. Happy Learning !!
In this article, we will discuss how to select and order multiple columns from a dataframe using pyspark in Python. For this, we are using sort () and orderBy () functions along with select () function. Select (): This method is used to select the part of dataframe columns and return a copy of that newly selected dataframe.
You can select the single or multiple columns of the DataFrame by passing the column names you wanted to select to the select () function. Since DataFrame is immutable, this creates a new DataFrame with selected columns. show () function is used to show the Dataframe contents.
val columnNames = Seq("col1","col2",....."coln") // using the string column names: val result = dataframe.select(columnNames.head, columnNames.tail: _*) // or, equivalently, using Column objects: val result = dataframe.select(columnNames.map(c => col(c)): _*) If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
