Is sharing cache/persisted dataframes between databricks notebook possible?

Question

I want to cache a table(dataframe) in one notebook and use it in another notebook , I am using same databricks cluster for both the notebooks.

Please suggest if this is possible , If yes then how ?

Answer 1

You can share dataframe between notebooks.

On the first notebook please register it as temp view:

df_shared.createOrReplaceGlobalTempView("df_shared")

On the second notebook please read it from global temp database:

global_temp_db = spark.conf.get("spark.sql.globalTempDatabase")
df_shared= table(global_temp_db + ".df_shared")

Answer 2

Yes it is possible based on the following setups .

You can register your dataframe as temp table . The lifetime of temp view created by createOrReplaceTempView() is tied to Spark Session in which the dataframe has been created.

spark.databricks.session.share to true

this setup global temporary views to share temporary views across notebooks. ref : link