Menu
I tried df.orderBy("col1").show(10) but it sorted in ascending order. df.sort("col1").show(10) also sorts in ascending order. I looked on stackoverflow and the answers I found were all outdated or referred to RDDs. I'd like to use the native dataframe in spark.
220
The Default sorting technique used by order by is ASC. We can import the PySpark function and used the DESC method to sort the data frame in Descending order. We can sort the elements by passing the columns within the Data Frame, the sorting can be done with one column to multiple column.
In Spark, you can use either sort() or orderBy() function of DataFrame/Dataset to sort by ascending or descending order based on single or multiple columns, you can also do sorting using Spark SQL sorting functions, In this article, I will explain all these different ways using Scala examples.
We can use either orderBy() or sort() method to sort the data in the dataframe. Pass asc() to sort the data in ascending order; otherwise, desc(). We can do this based on a single column or multiple columns.
In order to Rearrange or reorder the column in pyspark we will be using select function. To reorder the column in ascending order we will be using Sorted function. To reorder the column in descending order we will be using Sorted function with an argument reverse =True. We also rearrange the column by position.
You can also sort the column by importing the spark sql functions
import org.apache.spark.sql.functions._
df.orderBy(asc("col1"))
Or
import org.apache.spark.sql.functions._
df.sort(desc("col1"))
importing sqlContext.implicits._
import sqlContext.implicits._
df.orderBy($"col1".desc)
Or
import sqlContext.implicits._
df.sort($"col1".desc)
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With