How to delete all files from folder with Databricks dbutils

Question

Can someone let me know how to use the databricks dbutils to delete all files from a folder. I have tried the following but unfortunately, Databricks doesn't support wildcards.

dbutils.fs.rm('adl://azurelake.azuredatalakestore.net/landing/stageone/*')

Thanks

Answer 1

According to the documentation, the rm function receives 2 parameters :

rm(dir: String, recurse: boolean = false): boolean -> Removes a file or directory

Where the second parameter is a boolean flag to set the recursitivity, so you just need to set it to true:

dbutils.fs.rm('adl://azurelake.azuredatalakestore.net/landing/stageone/',True)

Answer 2

Something like this should work:

val PATH = "adl://azurelake.azuredatalakestore.net/landing/stageone/"
dbutils.fs.ls(PATH)
            .map(_.name)
            .foreach((file: String) => dbutils.fs.rm(PATH + file, true))