Apache Spark architecture

Question

Trying to find a complete documentation about an internal architecture of Apache Spark, but have no results there.

For example I'm trying to understand next thing: Assume that we have 1Tb text file on HDFS (3 nodes in a cluster, replication factor is 1). This file will be spitted into 128Mb chunks and each chunk will be stored only on one node. We run Spark Workers on these nodes. I know that Spark is trying to work with data stored in HDFS on the same node (to avoid network I/O). For example I'm trying to do a word count in this 1Tb text file.

Here I have next questions:

1. Does Spark will load chuck (128Mb) into RAM, count words, and then delete it from memory and do it sequentially? What if there will be no available RAM?
2. When does Spark will use not local data on HDFS?
3. What if I will need to do more complex task, when a results of each iteration on each Worker need to be transferred to all other Workers (shuffling?), do I need to write them by my self to HDFS and then read them? For example I can't understand how does K-means clustering or Gradient descent works on Spark.

I will appreciate any link to Apache Spark architecture guide.

Answer 1

Here are the answers to your questions

1. Spark will try to load 128Mb chunk into memory and process it in RAM. Keep in mind that that the size in memory can be several times larger than the original size of the raw file due to Java overhead (Java headers, etc). From my experience, it can be 2-4 time larger. If there is not enough memory (RAM) Spark will spill the data to local disk. You may want to tweak these two parameters to minimize the spill: spark.shuffle.memoryFraction and spark.storage.memoryFraction.

2. Spark will always try to use local data from in your HDFS. If the chunk not available locally it will retrieve it from another node in the cluster. more info

3. On shuffle, you do not need to manually save intermediate results to HDFS. Spark will write the results to local storage and shuffle only the data needed maximizing reuse of local storage for the next stage.

Here is good video that goes into more detail of Spark architecture, what happens during shuffle and tips for optimizing performance.

Answer 2

Adding to other answers, here I would like to include Spark core architecture diagram as it was mentioned in the question.

Master is entry point here.