Column alias after groupBy in pyspark

Question

I need the resulting data frame in the line below, to have an alias name "maxDiff" for the max('diff') column after groupBy. However, the below line does not makeany change, nor throw an error.

 grpdf = joined_df.groupBy(temp1.datestamp).max('diff').alias("maxDiff") 

Answer 1

You can use agg instead of calling max method:

from pyspark.sql.functions import max  joined_df.groupBy(temp1.datestamp).agg(max("diff").alias("maxDiff")) 

Similarly in Scala

import org.apache.spark.sql.functions.max  joined_df.groupBy($"datestamp").agg(max("diff").alias("maxDiff")) 

or

joined_df.groupBy($"datestamp").agg(max("diff").as("maxDiff")) 

Answer 2

This is because you are aliasing the whole DataFrame object, not Column. Here's an example how to alias the Column only:

import pyspark.sql.functions as func  grpdf = joined_df \     .groupBy(temp1.datestamp) \     .max('diff') \     .select(func.col("max(diff)").alias("maxDiff"))