Spark persist temp view

Question

I'm trying to persist a temp view with the purpose of querying it again via sql:

val df = spark.sqlContext.read.option("header", true).csv("xxx.csv")
df.createOrReplaceTempView("xxx")

persist/cache:

df.cache()                          // or
spark.sqlContext.cacheTable("xxx")  // or
df.persist(MEMORY_AND_DISK)         // or
spark.sql("CACHE TABLE xxx")

Then I move the underlying xxx.csv, and:

spark.sql("select * from xxx")

Upon which, I find that only CACHE TABLE xxx stores a copy. What am I doing wrong, how can persist eg. DISK_ONLY a queryable view/table?

Answer 1

First cache it, as df.cache, then register as df.createOrReplaceTempView("dfTEMP"), so now every time you will query dfTEMP such as val df1 = spark.sql("select * from dfTEMP) you will read it from memory (1st action on df1 will actually cache it), do not worry about persistence for now as if df does not fit into memory, i will spill the rest to disk.