Why does Databricks Connect Test not work on Mac?

Question

I've gone through the documentation for configuring databricks-connect but still get the error below when running databricks-connect test

Errors From Terminal

java.lang.NoSuchMethodError: org.apache.spark.internal.config.package$.STRING_REDACTION_PATTERN()Lorg/apache/spark/internal/config/ConfigEntry;
error: not found: value spark import spark.implicits._
error: not found: value spark import spark.sql
error: not found: value spark spark.range(100).reduce(_ + _)
Scala command failed to produce correct result

Steps Used to Setup Databricks Connect

• Created a cluster on Databricks with Runtime of 5.3 and Python 2
• Set Spark Config to spark.databricks.service.server.enabled true (Restarted)
• Created Python project using Pyenv and Python 2.7.15
• Removed pyspark pip uninstall pyspark
• Installed Databricks Connect pip install -U databricks-connect==5.3.*
• Configured databricks connect databricks-connect configure
  • Databricks Host: https://<account>.cloud.databricks.com
  • Databricks Token: <secret_token>
  • Cluster ID: <cluster_token>
  • Org ID: <left_blank>
  • Port: 15001

What is causing this error and is there a possible fix?

Used Resources

• Databricks DB Connect
• Databricks Connect Finally
• Why DB Connect doesn't work?

Answer 1

After further investigation, I found another StackOverflow post that discusses how the Java error hints to a version mismatch. I downgraded my Databricks cluster runtime to 5.1 and updated databricks-connect by running:

pip install -U databricks-connect==5.1.*

To validate, I ran:

databricks-connect test

Important items to remember:

1. Uninstall pyspark first because installing databricks-connect will install a slightly modified version of pyspark.
2. The databricks-connect version must match the cluster's DBR version.
3. The Python version of your local environment must match the Python version of your cluster. [E.g. 2.7, 3.5, 3.6, 3.7,...]