Apache Spark. High availability for spark client

Question

I am struggling to find any guides or manuals re high availability practices for client-side of spark applications. I was able to find recommendations for Spark master HA with ZooKeeper but that's different.

The problem is that if you run several instances of your application connecting to spark, you have to divide your available cluster resources between all of them which is an overkill.

Is there anything like the guide I'm looking for?

Answer 1

It depends on what your master is set to. If your using yarn-client High availability is free or some what free. If your running things in yarn-client or local mode if that machine goes down your pretty much done. Now really it boils down to what your trying to do. If you want compute resources separate from the hadoop data nodes I'd look into a mesos cluster. It's a great way to do sort of adhoc/Long running jobs without locking up yarn resources.

What is your data source and what are you trying to accomplish?