Spark job in Java: how to access files from 'resources' when run on a cluster

Question

I wrote a Spark job in Java. The job is packaged as a shaded jar and executed:

spark-submit my-jar.jar

In the code, there are some files (Freemarker templates) that reside in src/main/resources/templates. When run locally, I'm able access the files:

File[] files = new File("src/main/resources/templates/").listFiles();

When the job is run on a cluster, a null-pointer exception is returned when the previous line is executed.

If I run jar tf my-jar.jar I can see that the files are packaged in a templates/ folder:

 [...]
 templates/
 templates/my_template.ftl
 [...]

I'm just unable to read them; I suspect that .listFiles() tries to access the local filesystem on the cluster node, and the files aren't there.

I'm curious to know how I should package files to be used within a self-contained Spark job. I'd rather not copy them to HDFS outside of the job because it becomes messy to maintain.

Answer 1

I have accessed my resource file like below in spark-scala. I've share my code please check.

val fs=this.getClass().getClassLoader().getResourceAsStream("smoke_test/loadhadoop.txt")

val dataString=scala.io.Source.fromInputStream(fs).mkString