Run Databricks (Spark) Locally Using Docker Image

Question

I want to run databaricks/spark container locally. I've found an image below: https://github.com/databricks/containers

I've pulled it and run it. Docker ran successfully but I cannot figure out how to start cluster or launch UI or Notebook.

I've used standard image as it should have python notebooks: https://github.com/databricks/containers/tree/master/ubuntu/standard

Answer 1

First off, Databricks is more than Spark. Do you want to run Databricks or Spark ?

Databricks cannot be run on local docker containers. What you can do is to create a customer docker image with databricks code and then run it in cloud.

However, you may want to run jupyter pyspark-notebook in docker