Error Connecting to Databricks from local machine

Question

I am attempting to make a connection to Databricks from my Mac(Mojave).

I did a pip install -U databricks-connect==5.5.*

I start a spark-shell but when I try to query in spark I get the following error

Caused by: com.databricks.service.SparkServiceConnectionException: Invalid token

To connect to a Databricks cluster, you must specify an API token.
API Token: The API token used to confirm your identity to Databricks
  - Learn more about API tokens here: https://docs.databricks.com/api/latest/authentication.html#generate-a-token
  - Get current value: spark.conf.get("spark.databricks.service.token")
  - Set via conf: spark.conf.set("spark.databricks.service.token", <your API token>)
  - Set via environment variable: export DATABRICKS_API_TOKEN=<your API token>

I then run spark.conf.get("spark.databricks.service.token") and I get back the token I created. We tried using this token on a windows machine and it works.

Answer 1

Run:

databricks-connect configure

Follow the steps on the command prompt. You will be asked to enter your Databricks Token, enter your token, follow remaining steps and it should be resolved.

PS: When you are asked to enter your Databricks Host: provide a generic hostname for the region followed by your org-id. For example:

https://westeurope.azuredatabricks.net/?o=xxxx instead of https://adb-xxxx.xy.azuredatabricks.net/?o=xxxx