What is the difference between spark checkpoint and local checkpoint?

Question

What is the difference between spark checkpoint and local checkpoint? When making local checkpoint I see this in the spark UI:

It shows that local checkpoint is saved on memory.

Answer 1

Local checkpoint stores your data in executors storage (as shown in your screenshot). It is useful for truncating the lineage graph of an RDD, however, in case of node failure you will lose the data and you need to recompute it (depending on your application you may have to pay a high price).

'Standard' checkpoint stores your data in a reliable file system (like hdfs). It is more expensive to perform but you will not need to recompute the data even in case of failures. Of course, it truncates the lineage graph.

Truncating a long lineage graph avoid getting stack overflow exceptions and is particularly useful in iterative algorithms

Answer 2

• local checkpointing writes data in executors storage
• regular checkpointing writes data in HDFS

local checkpointing is faster than classic checkpointing but regular checkpointing is safer in that it leverages HDFS reliability like blocks replication.