list the files of a directory and subdirectory recursively in Databricks(DBFS)

Question

Using python/dbutils, how to display the files of the current directory & subdirectory recursively in Databricks file system(DBFS).

Answer 1

Surprising thing about dbutils.fs.ls (and %fs magic command) is that it doesn't seem to support any recursive switch. However, since ls function returns a list of FileInfo objects it's quite trivial to recursively iterate over them to get the whole content, e.g.:

def get_dir_content(ls_path):
  dir_paths = dbutils.fs.ls(ls_path)
  subdir_paths = [get_dir_content(p.path) for p in dir_paths if p.isDir() and p.path != ls_path]
  flat_subdir_paths = [p for subdir in subdir_paths for p in subdir]
  return list(map(lambda p: p.path, dir_paths)) + flat_subdir_paths
    

paths = get_dir_content('/databricks-datasets/COVID/CORD-19/2020-03-13')
[print(p) for p in paths]