What operations of spark is processed in parallel?

Question

I am trying to wrap my head about the whole concept of spark. I think I have a very rudimentary understanding about spark platform. From what I understand, Spark has the concept of RDDs which is a collection of "stuff" in memory so processing is faster. You transform RDDs by using methods like map and flatmaps. Since transformations are lazy, they are not processed until you call an action on the final RDD. What I am unclear about is, when you do an action, are the transformations run in parallel? Can you assign workers to do the action in parallel?

For example, lets say I have a text file that I load into an RDD,

lines = //loadRDD
lines.map(SomeFunction())
lines.count()

What is actually going on? Does SomeFunction() process a partition of the RDD? What is the parallel aspect?

Answer 1

lines is just a name for the RDD data structure resident in the driver which represents a partitioned list of rows. The partitions are being managed at each of your worker nodes when they are needed.

When your action count is called, Spark works backwards through the tasks to perform that action, resulting in a section of the file being read (a partition), SomeFunction being serialised and sent over the network to the workers, and executed on each row. If you have lots of workers then more than one partition can be read at a time and SomeFunction can be mapped over a partition for each worker/core.

Each worker sends the count of items for the partition it has processed back to the driver and the driver can sum up the counts from all the partitions and return the total.

Note: in your example, SomeFunction is redundant with respect to the count of items.