Databricks-Connect: Missing sparkContext

Question

I have issues using the newest version of databricks-connect (13.3.0). I would like to access the sparkContext and tried it as it worked for databricks-connect<13.0:

from databricks.connect import DatabricksSession
spark = DatabricksSession.builder.getOrCreate()
spark.sparkContext

However, now I get the error:

pyspark.errors.exceptions.base.PySparkNotImplementedError: [NOT_IMPLEMENTED] sparkContext() is not implemented.

Can someone help?

Answer 1

Databricks connect in versions 13+ is based on Spark Connect that doesn't support RDD APIs together with related objects like SparkContext. It's really documented as known limitation.

From Apache Docs for Spark Connect (Scala section):

APIs such as SparkContext and RDD are deprecated in all Spark Connect versions.

Spark Connect Docs (Apache)