Menu
I am running Spark Standalone cluster and on submitting the application, the spark driver stops with the following error.
16/01/12 23:26:14 INFO Worker: Asked to kill executor app-20160112232613-0012/0
16/01/12 23:26:14 INFO ExecutorRunner: Runner thread for executor app-20160112232613-0012/0 interrupted
16/01/12 23:26:14 INFO ExecutorRunner: Killing process!
16/01/12 23:26:14 ERROR FileAppender: Error writing stream to file /spark/spark-1.4.1/work/app-20160112232613-0012/0/stderr
java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70)
at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39)
at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772)
at org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38)
16/01/12 23:26:14 INFO Worker: Executor app-20160112232613-0012/0 finished with state KILLED exitStatus 143
16/01/12 23:26:14 INFO Worker: Cleaning up local directories for application app-20160112232613-0012
I am newbie to Spark and its processing.Please help me out in this.
217
What are the roles and responsibilities of worker nodes in the apache spark cluster? Is Worker Node in Spark is same as Slave Node? Worker node refers to node which runs the application code in the cluster. Worker Node is the Slave Node. Master node assign work and worker node actually perform the assigned tasks.
All nodes run services such as Node Agent and Yarn Node Manager. All worker nodes run the Spark Executor service. A Spark pool can be defined with node sizes that range from a Small compute node with 8 vCore and 64 GB of memory up to a XXLarge compute node with 64 vCore and 432 GB of memory per node.
Spark Standalone has 2 parts, the first is configuring the resources for the Worker, the second is the resource allocation for a specific application. The user must configure the Workers to have a set of resources available so that it can assign them out to Executors.
The worker consists of processes that can run in parallel to perform the tasks scheduled by the driver program. These processes are called executors. Whenever a client runs the application code, the driver programs instantiates Spark Context, converts the transformations and actions into logical DAG of execution.
The error is not caused by the java.io.IOException since you can see clearly 16/01/12 23:26:14 INFO Worker: Asked to kill executor app-20160112232613-0012/0. This exception is raised afterwards when spark tries to write the log files, in which you will also observe the cause of your error.
Even if you run with root privilege spark-submit, it is the spark user that writes the files. I am guessing you are running on your laptop this. Try running sudo chmod -R 777 on your spark folder.
170
In my case, the problem was, spark driver was not able to get the dependencies from the executable jar that was submitted. Merged all dependencies and converted them to a single executable. It fixed the issue.
Please bear with my terminologies :)
29
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
