Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

apache spark agg( ) function

For a sample dataframe scholor,

scala> scholor.show

| id|  name|age|sal|base|

For above, both below, gives same output. What will be use of agg() then. It just for name.

scala> scholor.groupBy("age").sum("base").show      /*with out agg */

scala> scholor.groupBy("age").agg(sum("base")).show        /* with agg */
+---+---------+
|age|sum(base)|
+---+---------+

Does agg() need any varargs as arguments? What is need of agg()?

Thanks in advance.

like image 455
Raghav Avatar asked Jan 05 '23 05:01

Raghav


1 Answers

In order write .sum this method has to exist. It is hardcoded on the API. Using .agg you can provide other aggregating functions, sum("column") is just one of them.

like image 174
V-Lamp Avatar answered Jan 06 '23 19:01

V-Lamp