How to set Spark job staging location

Question

My spark job is failing because the user doesn't have access to directory where spark is trying to write staging or temp dataset.

2017-03-10 10:25:47,0928 ERROR JniCommon fs/client/fileclient/cc/jni_MapRClient.cc:2072 Thread: 26413 mkdirs failed for /user/cxpdiprod/.sparkStaging/application_1488190062017_14041, error 13 Exception in thread "main" org.apache.hadoop.security.AccessControlException: User cxpdiprod(user id 99871) has been denied access to create application_1488190062017_14041 at com.mapr.fs.MapRFileSystem.makeDir(MapRFileSystem.java:1250) at com.mapr.fs.MapRFileSystem.mkdirs(MapRFileSystem.java:1270) at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1913) at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:634) at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:356) at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:727) at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:142) at org.apache.spark.deploy.yarn.Client.run(Client.scala:1021) at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1081) at org.apache.spark.deploy.yarn.Client.main(Client.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:742)

The user 'cxpdiprod' has access to some other directory in cluster i.e. /mapr/ui/abc. Are there any property to set the different directory for temp and staging files?

Answer 1

Add this property spark.yarn.stagingDir with the desired staging location in spark-defaults.conf. By default, the staging location is current user's home directory /user/username/ in HDFS filesystem.