What is Databricks Spark cluster manager? Can it be changed?

Question

Original Spark distributive supports several cluster managers like YARN, Mesos, Spark Standalone, K8s. I can't find what is under the hood in Databricks Spark, which cluster manager it is using, and is it possible to change?

What's Databricks Spark architecture?

Thanks.

Answer 1

You can't change the cluster manager in Databricks, and you really don't need that because this part is managed for you. You can think about it as a kind of standalone cluster, but there are differences. General Databricks architecture is shown here.

You can change the cluster configuration by different means - init scripts, configuration parameters, etc. See documentation for more details.