I can't understand 'RDD.map{ case (A, B) => A } ' in Scala Spark

Question

I'm relatively new to Scala Spark. I've got a question with map method.

My understanding: map is a RDD method, it accepts a function as its parameter like:map(line => line.split(","))

I found really hard to understand this kind of expression.

val uniqueUsers = data.map { case (user, product, price) => user }.distinct().count()

Could someone explain two things for me:

1. why {} is used not ()
2. can I regard case (user, product, price) => user as a function? If not, what is it?

Thank you in advance.

Answer 1

In Scala the syntax { case arg => body } is a Partial Function.

definition of Partial Function from Scala-Doc

A partial function of type PartialFunction[A, B] is a unary function where the domain does not necessarily include all values of type A. The function isDefinedAt allows to test dynamically if a value is in the domain of the function.

In your case of { case (user, product, price) => user } you have defined a Partial Function that a takes a Tuple3 object as input. This Tuple3 object is unpacked as 3 variables of user, product and price and the function body just returns the user.

so to answer your questions

1. why {} is used not ()

   because partial functions have to be wrapped by curly braces.

2. an I regard case (user, product, price) => user as a function? If not, what is it?

   yes. { case (user, product, price) => user } is a special type of Function called PartialFunction which is defined only for specific inputs and is not defined for other inputs. In your case the PartialFunction is defined only for input of Tuple3[T1,T2,T3] where T1,T2, and T3 are types of user,product and price objects